{"id":6143,"date":"2015-06-17T12:00:17","date_gmt":"2015-06-17T10:00:17","guid":{"rendered":"http:\/\/www.centigrade.de\/blog\/?p=6143"},"modified":"2020-01-25T19:39:41","modified_gmt":"2020-01-25T18:39:41","slug":"contextual-user-interface-design-engineering-with-the-universal-windows-platform","status":"publish","type":"blog","link":"https:\/\/www.centigrade.de\/en\/blog\/contextual-user-interface-design-engineering-with-the-universal-windows-platform\/","title":{"rendered":"Contextual User Interface Design Engineering with the Universal Windows Platform"},"content":{"rendered":"

Recently I gave a talk at the dotnet Cologne<\/a>\u00a0and also at the DWX \u00a0Developer Week<\/a> titled “4K and other challenges \u2013 Next Generation Desktop UIs for Windows 10”. The session discussed the term Universal App Platform in Windows 10 and showed what a developer can make out of it in order to create future oriented user interfaces. This blog article is not only supposed to target those who attended my session, but also those who were not present to hear it. Moreover the article will provide further information to the topic. As in the session there will be a coding part at the end where some new Universal App features are shown.<\/p>\n

Retrospection of Windows 8<\/h3>\n
\"Windows

Windows 8 Start screen (Reference: Microsoft News Center<\/a>)<\/p><\/div>\n

One UI to rule them all<\/em> may have been the first impression Windows 8 made to most of its users back in 2012. The touch oriented, flashy tile interface and the missing start menu seemed to be a major break in consistency for many users. However there was a lot going on behind the scenes. In accordance to the Platform Convergence Journey<\/a> the windows kernel was unified for Xbox, Windows 8, and Windows Phone 8 with introduction of Windows 8. Windows 8.1 then brought together the app model. Even then, under the term Universal App, it was possible \u2013 although a bit cumbersome \u2013 to develop a Windows Store App with a single app identity and the same codebase for desktops and smartphones.<\/p>\n

\"Platform

Platform Convergence Journey (Reference: Screencapture Microsoft Virtual Academy<\/a>)<\/p><\/div>\n

The Windows 10 Philosophy<\/h3>\n
\"Windows

Windows 10 desktop and start menu (Reference: Microsoft News Center<\/a>)<\/p><\/div>\n

At first sight it appears as though Windows 10 takes a step backwards. From the viewpoint of a desktop user that may be right. Windows Store Apps are able to run in a window, there is a classical start menu in combination with some tiles, and the touchscreen interaction is not assumed as the default. The UI is only adapted to this if Windows is running in tablet mode. But from the point of view of a developer Microsoft is consequently targeting the unification: in future there will be no separation of Windows versions for different devices anymore. The Windows 10<\/em> will run on all devices, be it a smartphone, desktop, Xbox, Surface Hub, or even HoloLens or IoT devices like the RaspberryPi. On the so-called Universal Windows Platform<\/a> a future Windows App will likewise run on all devices \u2013 like in the true meaning of write once, deploy everywhere<\/em>.<\/p>\n

\"Universal

Universal Windows Platform (Reference: Microsoft Blog<\/a>)<\/p><\/div>\n

Hence the developer will create one app \u2013 a truly, single binary \u2013 which will be deployed and run across all devices. However an application should by no means appear the same on all devices. In fact the developers are supposed to adapt the UI for the different devices, be it a smartwatch, a mobile line-of-business app, a desktop application or a UI for controlling a refrigerator in the internet of things. By using the MVVM Pattern<\/a> a clean separation of concerns is feasible. Because of that the internal business logic can be isolated from the interchangeable, customizable and highly optimized presentation layer.<\/p>\n

\"MVVM:

MVVM: Model-View-ViewModel software architecture<\/p><\/div>\n

Adaptive UIs<\/h3>\n

What does such a customization look like? What has to be adjusted for different devices and where? Which tools are available to the developer to adjust the UI? How can different device families technically be filtered and customizations made according to this information?<\/p>\n

The most intuitive component when thinking about different devices may be the form factors, like display size, resolution and the resulting physical \u2013 that is the one given by the device \u2013 pixel density. And in connection with that the distance between the user and the device \u2013 the view distance. To shortly summarize this: a higher physical resolution<\/a> on a smaller physical diagonal display size results in a higher pixel density also known as PPI (pixel per inch). This is even more important when the viewer is very close to the device. Therefore high density displays for smartphones are rather the rule than the exception. High PPI can be used to either fit more content into the same space \u2013 which only makes sense to a certain degree \u2013 or to display the same content in higher quality. Circular shapes, that are always approximations on digital devices, will come closer to their ideal representation. The user interface of the application will appear crisp and smooth to the user.<\/p>\n

Often it is not enough to only focus on these factors. Coming from the web development, Responsive Design<\/em><\/a> can be used to differentiate between desktops and smartphones based on the resolution. A two column layout will be made into a one column layout, only the most relevant content will be displayed or moved to second levels of navigation. This is a first step that quickly reaches its limits. Responsive Design is a rather technical consideration that only reacts to device parameters. If the specific user role and the usage scenario is also considered during the UX design process, one will achieve remarkably better, and more intuitive results. This primary consideration of the user is called Contextual Design<\/em>.<\/p>\n

Contextual Design<\/h3>\n
\"Microsoft

Microsoft HoloLens (Reference: Microsoft News Center<\/a>)<\/p><\/div>\n

Virtual reality and augmented reality applications are the best examples to illustrate this, since they greatly benefit from Contextual Design. Because, what does the term resolution mean for this device family? A high resolution for a VR-glass is in the first place important in order to avoid the so-called Screen-door effect<\/em><\/a>. This is due to the small distance between the display and the user\u2019s eye. However, what is even more important is where the UI is placed inside the 3D scene of the VR- or AR-application, because this will have an impact on the perceived resolution. UI elements could thereby be placed in the real world like recently demonstrated in Microsoft\u2019s keynote talk of the Build conference 2015<\/a>. It is hard to tackle this challenge with classical Responsive Design.<\/p>\n

\"Microsoft

Microsoft HoloLens: Concept for an augmented reality application (Reference: Microsoft News Center<\/a>)<\/p><\/div>\n

Though apart from VR or AR, it is already sufficient to simply take a look at the desktop \u2013 meaning a stationary computer \u2013 be it an office pc, a conference monitor or a huge screen in a space station’s control room. In these situations one will quickly end up in scenarios where Responsive Design is not enough.<\/p>\n

\"Control

Control room of the esa space station (Reference: esa Homepage<\/a>)<\/p><\/div>\n

Also, there are some developers who work with their screens in portrait mode. Are there applications that react to whether the monitor is rotated 90 degrees? Pair-programming sessions or code reviews in a walkthrough manner include multiple persons in front of a single display. Some people are closer to the screen than others. Which applications take these scenarios into consideration?<\/p>\n

\"A

A typical desktop?<\/p><\/div>\n

Microsoft’s Surface Hub<\/a> is a huge 84 inches conference screen with multiple cameras, directed microphones and pen input. The presenter stands right beside the screen and can write directly onto the application, while viewers sit in quite a distance away, and some attendees are only connected via remote while still seeing the same content on their local computer. So far, I have not come across any meeting application that is \u2013 without major adjustments \u2013 optimized for this scenario.<\/p>\n

\"Microsoft

Microsoft Surface Hub in meetings (Reference: Microsoft News Center<\/a>)<\/p><\/div>\n

Universal Apps targeting a broad audience will be faced with these kinds of challenges, which can only be solved by Contextual UI Design. In doing so, an application will know its usage scenario and will be able to react accordingly. The view distance can e.g. be detected through eye-tracking sensors on the device. An app will know whether it is in portrait mode and will be able to optimize its content for this scenario. Only if a device has a microphone as input type, the push-to-talk functionality will be visible. There are numerous challenges that can only be solved properly if the user is taken into account. From a technical perspective it seems like very hard additional work. Processing eye-tracking information and reacting with layout and content optimizations seem to be a herculean task. However Windows 10, Visual Studio 2015 and the Universal App Platform support the developer with new features to create context sensitive software with great user experience. We will now focus on some of these tools.<\/p>\n

UWP Features in Windows 10<\/h3>\n

Like the device independent pixels<\/a> in WPF, Microsoft continues to focus on a term for abstract pixel information and introduces so-called effective pixels<\/em><\/a>. These are based on the pixel density of the device and a heuristic view distance. It results in abstract pixel information, so that a developer has to care less about physical details. Elements appear<\/em> the same size to the user across all devices.<\/p>\n

Furthermore the RelativePanel<\/a> is a new layout container intended to structure elements inside this container in relation to each other. This is extremely powerful when the layout is supposed to change and adapt dynamically. Prior to that you had to implement a custom panel or make complex adjustments to a Grid.<\/p>\n

In order to react to different device families and form factors, we will now take a look at the so-called VisualState-Triggers<\/a>. Microsoft proposes to use the VisualStateManager and VisualStates for different view states.<\/p>\n

Implementation sample: A custom VisualState-Trigger<\/h2>\n

Let\u2019s imagine the following scenario: The user owns a monitor which is able to measure the current view distance (e.g. via webcam, a head tracking system, infrared, the Kinect etc.). As a developer I want my application to adjust its font size according to the view distance. The font size should always appear the same size to the users. They could be very close to the screen (because they may take notes with a pen on a touch sensitive display), in a normal view distance, or rather further away while having a pair-session. As a UI developer I have to technically detect these changes to finally respond to them in XAML in an easy way. We will now implement this while not focusing on crafting a beautiful and complete application. In fact, we will focus on demonstrating the newly available features of the platform.<\/p>\n

First of all we will create a new Windows Universal App<\/em> project, open the MainPage.xaml<\/span> and place some text in it. At the same time we introduce two visual states as a showcase for the near and far view distance:
\n<\/code><\/p>\n

<<\/span>Grid<\/span> Margin<\/span>=\"25\"<\/span> Background<\/span>=\"White\"<\/span>><\/span>\r\n   <<\/span>VisualStateManager.VisualStateGroups<\/span>><\/span>\r\n      <<\/span>VisualStateGroup<\/span> x:Name<\/span>=\"CommonStates\"<\/span>><\/span>\r\n         <<\/span>VisualState<\/span> x:Name<\/span>=\"Near\"<\/span>><\/span>\r\n            <!-- Trigger and reaction missing for now. --><\/span>\r\n         <\/<\/span>VisualState<\/span>><\/span>\r\n         <!-- Here could be more states \u2026 --><\/span>\r\n         <<\/span>VisualState<\/span> x:Name<\/span>=\"Far\"<\/span>><\/span>\r\n            <!-- Trigger and reaction missing for now. --><\/span>\r\n         <\/<\/span>VisualState<\/span>><\/span>\r\n      <\/<\/span>VisualStateGroup<\/span>><\/span>\r\n   <\/<\/span>VisualStateManager.VisualStateGroups<\/span>><\/span>\r\n   <<\/span>TextBlock<\/span> x:Name<\/span>=\"Output\"<\/span>\r\n      FontSize<\/span>=\"20\"<\/span>\r\n      Text<\/span>=\"Lorem ipsum dolor sit amet, consetetur\u2026\"<\/span>\r\n      TextTrimming<\/span>=\"CharacterEllipsis\"<\/span>\r\n      TextWrapping<\/span>=\"Wrap\"<\/span> \/><\/span>\r\n<\/<\/span>Grid<\/span>><\/span>\r\n<\/pre>\n
 <\/p>\n
<<\/span>VisualState<\/span> x:Name<\/span>=\"Near\"<\/span>><\/span>\r\n   <<\/span>VisualState.StateTriggers<\/span>><\/span>\r\n      <<\/span>triggers:ViewDistanceTrigger<\/span> MaxViewDistance<\/span>=\"12\"<\/span> \/><\/span>\r\n   <\/<\/span>VisualState.StateTriggers<\/span>><\/span>\r\n   <!-- What to do? --><\/span>\r\n<\/<\/span>VisualState<\/span>><\/span>\r\n<\/pre>\n
 <\/p>\n
public<\/span> class<\/span> ViewDistanceTrigger : StateTriggerBase\r\n{\r\n   public<\/span> double<\/span> MinViewDistance { get<\/span>; set<\/span>; } = 0;\r\n   public<\/span> double<\/span> MaxViewDistance { get<\/span>; set<\/span>; } = double<\/span>.PositiveInfinity;\r\n\r\n   private<\/span> IViewDistanceDetector viewDistanceDetector;\r\n\r\n   public<\/span> ViewDistanceTrigger()\r\n   {\r\n      viewDistanceDetector = new ViewDistanceDetector();\r\n      viewDistanceDetector.ViewDistanceChanged +=\r\n\t     ViewDistanceDetector_ViewDistanceChanged;\r\n   }\r\n\r\n   private<\/span> void ViewDistanceDetector_ViewDistanceChanged\r\n      (object<\/span> sender, ViewDistanceChangedEventArgs eventArgs)\r\n   {\r\n      double<\/span> currentViewDistance = eventArgs.ViewDistanceInInch;\r\n   \r\n      if<\/span>((currentViewDistance >= MinViewDistance)\r\n\t     && (currentViewDistance <= MaxViewDistance))\r\n      {\r\n         SetActive(true<\/span>);\r\n      }\r\n      else<\/span>\r\n      {\r\n         SetActive(false<\/span>);\r\n      }\r\n   }\r\n}\r\n<\/pre>\n
 <\/p>\n
<<\/span>VisualState<\/span> x:Name<\/span>=\"Near\"<\/span>><\/span>\r\n   <<\/span>VisualState.StateTriggers<\/span>><\/span>\r\n      <<\/span>triggers:ViewDistanceTrigger<\/span> MaxViewDistance<\/span>=\"12\"<\/span> \/><\/span>\r\n   <\/<\/span>VisualState.StateTriggers<\/span>><\/span>\r\n   <<\/span>VisualState.Setters<\/span>><\/span>\r\n      <<\/span>Setter<\/span> \r\n         Target<\/span>=\"Output.FontSize\"<\/span>\r\n         Value<\/span>=\"{StaticResource FontSize.Small}\"<\/span> \/><\/span>\r\n   <\/<\/span>VisualState.Setters<\/span>><\/span>\r\n<\/<\/span>VisualState<\/span>><\/span>\r\n<<\/span>VisualState<\/span> x:Name<\/span>=\"Far\"<\/span>><\/span>\r\n   <<\/span>VisualState.StateTriggers<\/span>><\/span>\r\n      <<\/span>triggers:ViewDistanceTrigger<\/span> MaxViewDistance<\/span>=\"25\"<\/span> \/><\/span>\r\n   <\/<\/span>VisualState.StateTriggers<\/span>><\/span>\r\n   <<\/span>VisualState.Setters<\/span>><\/span>\r\n      <<\/span>Setter<\/span> \r\n         Target<\/span>=\"Output.FontSize\"<\/span>\r\n         Value<\/span>=\"{StaticResource FontSize.Large}\"<\/span> \/><\/span>\r\n   <\/<\/span>VisualState.Setters<\/span>><\/span>\r\n<\/<\/span>VisualState<\/span>><\/span>\r\n<\/pre>\n
 <\/p>\n
Of course an animated, smooth transition and more states inbetween would be nice. However we only wanted to show how to write custom triggers in a Universal App to make adaptive and context sensitive changes to the UI in a compact way. It is clear that this concept can be expanded to a variety of contextual information: is the screen in portrait mode? Do I have a microphone for speech input? Is an Xbox controller conntected to my device? Is my application running on a specific device family? All this information could comfortably be stored inside triggers. It is to be hoped that Microsoft will provide more built-in triggers that will release the developer from the burden to write his own ones.<\/p>\n
Conclusion<\/h3>\n
As we have just seen, Windows 10 and the Universal Windows Platform will provide us with a couple of aids to optimize the UI for different devices. Microsoft is responsible for doing that as they promise that a single app will run across all devices. Having that said, these tools can also be utilized to enhance the experience of desktop applications. Insights we gained while developing smartphone UIs for different user scenarios and roles can be transformed to the desktop world. Even in fixed desktop environments there is a variety of contexts and diverse possibilities for interaction. We have to react to all those factors in a proper way. Concerning the UX design the desktop hasn\u2019t come to an end yet. There still is a wide range of innovative concepts hidden that should be explored and eventually used to create astonishing applications.<\/p>\n
Download<\/h3>\n
If you would like to take a closer look at\u00a0my presentation slides, code samples\u00a0from the DWX or code snippets from this blog, you can do so:<\/p>\n
To keep it simple, we will only change the font size of our TextBlock. With help from the new VisualState setters<\/a> it is easy to apply discrete changes to the values without a complex definition of Storyboards. This feature is very handy for initial state changes:
\n<\/code><\/p>\n
This trigger must now be implemented. Therefore you can inherit from StateTriggerBase<\/a><\/span> and use the method SetActive<\/span> to (de)active the trigger. The VisualState defined in XAML becomes active as soon as SetActive(true)<\/span> is called. The trigger could for instance use a component called IViewDistanceDetector<\/span>, which is responsible for detecting change of the view distance:
\n<\/code><\/p>\n
Currently Microsoft only ships the AdaptiveTrigger<\/a><\/span>. This trigger can only react to the screen size (in effective pixels). So, at the moment there is no built-in feature to fulfill our requirements. Fortunately we can write our own trigger and use it in XAML. Let’s look at the following example: as soon as the view distance is less than 12 inches we want to switch to our near state. We can represent this as a trigger in the following way:
\n<\/code><\/p>\n