Monday, December 14, 2015

Custom SPSD Extension - Extending User Profiles with Custom Properties


Earlier this month, I wrote about using SPSD to manage SharePoint custom solutions automation and using that tool to manage the Application Life Cycle management as well. I talked about writing custom extensions as per your need.

In this article I’ll talk about one such extension which is used to create new User Profile Properties that may be required by your customization on SharePoint platform. Again, I expect that you are familiar with capabilities of SPSD and you have read the documentation of SPSD before reading further. However, if you are have not created your own extension before or want to use one of the existing extension, this blog may serve as a reference for you. Feel free to leave comments if steps are unclear.

My goal is to create an SPSD extension which extends user profiles with custom properties but I don’t want to hard-wire the properties in the script. I want to use SPSD’s ability to pass XML when an extension executes and want to write a script (extension) that reads the chunk of XML (user profile specifications) and creates the user profile properties as specified. The beauty of this approach is that I can use this SPSD extension in many many other projects where I can use SPSD for deployments.

The figure below are high level steps. Steps in GREEN color are related to creating extension and specifying when to call the extension in the deployment process. Steps in ORANGE explains what happens when the SPSD is run for deployment. Again there are many things that happen when SPSD’s deploy.bat is run but scope of the figure below is the extension itself.



Here are the steps to create a new extension, define an instance of it and call the instance from {machineName}.xml or the default.xml.

1a. Create a folder under \Scripts\Extensions folder as shown below and create a .ps1 file and implement logic to create custom user profile properties. As a convention, name of the folder shall be same as name of the .ps1 and extension name. At minimum, you would create one PowerShell function, the signature of which is governed by SPSD. Copy of the code is given below.

function Create-CustomUserProfileProperties($parameters, [System.Xml.XmlElement]$data, [string]$extId, [string]$extensionPath) {
   
[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.Office.Server")
   
[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.Office.Server.UserProfiles")

  

   
if ($data -eq $null) {
       
Log -message "No Properties are specified in the configuration. User Profile Properties will not be created." -type $SPSD.LogTypes.Warning
       
return;
   
}
   
   
#$site = Get-SPSite $vars["SiteUrl"] # Get-SPSite   
   
#$context = Get-SPServiceContext $site # Get The service Context
   
   
$profileApp = @(Get-SPServiceApplication | ? {$_.TypeName -eq $vars["UserProfileApplicationName"]})[0]
    
$context= [Microsoft.SharePoint.SPServiceContext]::GetContext($profileApp.ServiceApplicationProxyGroup, [Microsoft.SharePoint.SPSiteSubscriptionIdentifier]::Default)

   
$context
   
$upcm = New-Object Microsoft.Office.Server.UserProfiles.UserProfileConfigManager($context);     
   
$ppm = $upcm.ProfilePropertyManager
   
$cpm = $ppm.GetCoreProperties()
   
$ptpm = $ppm.GetProfileTypeProperties([Microsoft.Office.Server.UserProfiles.ProfileType]::User)
   
$psm = [Microsoft.Office.Server.UserProfiles.ProfileSubTypeManager]::Get($context)
   
$ps = $psm.GetProfileSubtype([Microsoft.Office.Server.UserProfiles.ProfileSubtypeManager]::GetDefaultProfileName( [Microsoft.Office.Server.UserProfiles.ProfileType]::User))
   
$pspm = $ps.Properties

   
   
foreach($prop in $data.SelectNodes("//Property")) {
       
$property = $pspm.GetPropertyByName($prop.Name)
       
if ($property -eq $null)
       
{
           
$DefaultPrivacy=$prop.DefaultPrivacy
           
$PrivacyPolicy=$prop.PrivacyPolicy
           
$coreProp = $cpm.Create($false)
           
$coreProp.Name = $prop.Name
           
$coreProp.DisplayName = $prop.DisplayName
           
$coreProp.Type = $prop.Type
           
$coreProp.Length = [Int]::Parse( $prop.Length)
           
$cpm.Add($coreProp)
           
$profileTypeProp = $ptpm.Create($coreProp);
           
$profileTypeProp.IsVisibleOnEditor = $prop.IsVisibleOnEditor;           
           
$profileTypeProp.IsVisibleOnViewer = $prop.IsVisibleOnViewer;
           
$ptpm.Add($profileTypeProp)
           
$profileSubTypeProp = $pspm.Create($profileTypeProp);
           
$profileSubTypeProp.DefaultPrivacy = [Microsoft.Office.Server.UserProfiles.Privacy]::$DefaultPrivacy
           
$profileSubTypeProp.PrivacyPolicy = [Microsoft.Office.Server.UserProfiles.PrivacyPolicy]::$PrivacyPolicy         
           
$pspm.Add($profileSubTypeProp)
           
Log -message ([String]::Format( "Profile Property {0} was created successfully.",$prop.Name )) -type $SPSD.LogTypes.Normal
       
}
       
else
       
{
           
Log -message ([String]::Format( "Profile Property {0} already exist. Skipped.", $prop.Name)) -type $SPSD.LogTypes.Warning
       
}
       
   
}

   
}


1b. Define a manifest.xml file, the structure of which is governed by SPSD. Basically, it defines the Type (a logical name of extension – by convention, it can be same as folder name), description and name of the .ps1 file that we created in above step.

<?xml version="1.0" encoding="utf-8" ?>
<SPSD>
 
<Extensions>
   
<Extension>
       
<Manifest>
           
<Type>UserProfileCustomProperty</Type>
           
<Description>
               Use this extension to create custom user profile properties
           
</Description>
           
<Version>1.0.0.0</Version>
           
<Script>CustomUserProPropsUserProfileCustomProperties.ps1</Script>
           
<Enabled>true</Enabled>
       
</Manifest>

   
</Extension>
  
</Extensions>
</SPSD>



2. Under \Environments\Extensions folder, create an XML file as per the structure expected by SPSD – this becomes an instance of extension. Basically, here you will specify ID of instance, Type of extension, Event to specify when should this extension instance be called at run time (For instance AfterDeploy, BeforeDeploy), parameters and Data. The Parameters you specify here are available as an input. Under the Data Node you can specify an arbitrary XML which your PowerShell script can understand. Example shown below the figure.

<?xml version="1.0" encoding="utf-8" ?>
<SPSD Version="5.0.1.6438">
 
<Extensions>
   
<Extension ID="CreateUserProfileProperties" Type="CustomUserProProps" >
     
<Events>
       
<Event Name="AfterDeploy">Create-CustomUserProfileProperties</Event>
     
</Events>
     
<Parameters>
       
<Parameter Name="SomeParam">Some Value</Parameter>
     
</Parameters>
     
<Data>
       
<!--
        DefaultPrivacy - Values
          "Public"    Gives everyone visibility to the user's profile properties and other My Site content.
          "Contacts"    This object, member, or enumeration is deprecated and is not intended to be used in your code. Limits the visibility of users' profile properties, and other My Site content, to my colleagues.
          "Organization"    This object, member, or enumeration is deprecated and is not intended to be used in your code. Limits the visibility of users' profile properties, and other My Site content, to my workgroup.
          "Manager"    This object, member, or enumeration is deprecated and is not intended to be used in your code. Limits the visibility of users' profile properties, and other My Site content, to my manager and me.
          "Private"    Limits the visibility of the user's profile properties and other My Site content to the user only.
          "NotSet"    The privacy level for user's profile properties and other My Site content is not set.
         
         
          PrivacyPolicy - Values
          "Mandatory"-    Makes it a requirement that the user fill in a value.
          "OptIn"    Opt-in to provide a privacy policy value for a property.
          "OptOut"    Opt-out from providing a privacy policy value for a property.
          "Disabled"    Turns off the feature and hides all related user interface.
         
          Length - Max 3600
        -->
       
<Property Name="YourPropName" DisplayName="Prop DisplayName" Type="string" Length="3600" DefaultPrivacy="Private" PrivacyPolicy="OptIn" IsVisibleOnEditor="OptIn" IsVisibleOnViewer="$true"></Property>
      
     
</Data>
   
</Extension>
 
</Extensions>
</SPSD>



3. Modify {TargetMachineName}.xml or “Default.xml” file under \Environments folder and provide a reference to above extension instance file. Example shown below the figure.

<Extensions ID="Default">

       
<!-- AFTER WSP DEPLOYMENTS-->
       
   
<Extension ID="CreateUserProfileProperties" Type="CustomUserProProps" FilePath="Extensions\CustomUPProps.xml" />
   
       
</
Extensions>



Now, let’s understand what happens at runtime, at this time you should be able to connect the dots (steps in orange color above):

R1. When SPSD’s batch file (typically deploy.bat) file is run, it will read the {targetmachine}.xml or default.xml (if machine specific file is absent) and start to do whatever is specified in there, for reference check SPSD’s documentation. At this point SPSD would make an inventory of things it needs to do and determine the order in which it should execute the deployment.

R2. Whenever the time comes for our extension (as dictated by step 2 and 3 above), SPSD will figure out which method of the PowerShell script to call and what data to pass (as defined in step 2 above)

R3. Finally, the extension is executed by SPSD and messages are recorded for reference purpose.






That’s it for now! Feel free to leave comments.

Tuesday, December 8, 2015

AngularJS: Building a Postal Lookup Directive using Google APIs

In this post, I will show you how to build an angular directive that can be used to validate and provide full US addresses to the view in which it is used. Let’s see in action first:

SNAGHTMLb452191

So on right side, I have a text box – as you start typing in the address, it uses Google’s geo-coding APIs behind the scene and uses the Angular Bootstrap - Type Ahead directive to display the suggestions. Once the address is selected, it shows the full Address and a “Copy Address” Button as shown below:

image

When a copy button is clicked, the directive calls a function that was provided with the directive with address object. The consumer view (left side above) and use that and display in the fields. So as you can imagine, the directive discussed here is just a helping user to improve experience of selecting an address.

Let’s understand the dependencies:

1. Google API should have been included:

<script type="text/javascript" src="https://maps.googleapis.com/maps/api/js?v=3.exp&sensor=false"></script>



2. The Type-ahead depends on UI Bootstrap


3. All other Angular Plumbing should have been done, the purpose of this article is to show the angular directive code only


So my angular directive looks like this:


   1:  (function() {
   2:      'use strict';
   3:   
   4:      angular
   5:          .module('app')
   6:          .directive('stPostalLookup', stPostalLookup);
   7:   
   8:      stPostalLookup.$inject = ['$window'];
   9:      
  10:      function stPostalLookup ($window) {
  11:          // Usage:
  12:          //     <stPostalLookup></stPostalLookup>
  13:          // Creates:
  14:          // 
  15:          var directive = {
  16:              link: link,
  17:              restrict: 'EA',
  18:              controller: 'stPostalLookupController',
  19:              scope:{
  20:                  addressCallback: "&"
  22:              },
  23:              //controllerAs: 'vm',
  24:              templateUrl: function (element) {
  25:                  return '/app/components/postal-lookup.html';
  26:              }
  27:          };
  28:          return directive;
  29:   
  30:          function link(scope, element, attrs) {
  31:          }
  32:      }
  33:   
  34:   
  35:      angular
  36:          .module('app')
  37:          .controller('stPostalLookupController', ['$scope','$http','$q','common','config', stPostalLookupController]);
  38:   
  39:      function stPostalLookupController($scope, $http, $q, common, config) {
  40:          //var vm = this;
  41:          $scope.address = undefined;
  42:          $scope.getLocation = getLocation;
  43:          $scope.provideAddress = provideAddress;
  44:   
  45:          function getLocation(val) {
  46:   
  47:              var geocoder = new google.maps.Geocoder();
  48:              var address = val;
  49:   
  50:              var deferred = $q.defer();
  51:   
  52:              if (geocoder) {
  53:                  geocoder.geocode({ 'address': address }, function (results, status) {
  54:                      if (status == google.maps.GeocoderStatus.OK) {
  55:                          var data = Enumerable.From(results)
  56:                                      .Where(function (x) {
  57:                                          return x.address_components && x.address_components.length > 4 && x.formatted_address
  58:                                      })
  59:                                      .Select(function (x) {
  60:                                          var streetNo = _extractStreetNo(x.address_components);
  61:                                          var streetAdreess = _extractStreetAddress(x.address_components);
  62:                                          if (!angular.isDefined(streetAdreess)) {
  63:                                              streetAdreess = '';
  64:                                          }
  65:                                          if (!angular.isDefined(streetNo)) {
  66:                                              streetNo = '';
  67:                                          }
  68:                                          return {
  69:                                              title: x.formatted_address,
  70:                                              city: _extractCity(x.address_components),
  71:                                              state: _extractState(x.address_components),
  72:                                              streetAddress: sprintf("%(streetNumber)s %(streetAddress)s", { streetNumber: streetNo, streetAddress: streetAdreess }),
  73:                                              country: _extractCountry(x.address_components),
  74:                                              postalCode: _extractPostalCode(x.address_components)
  75:                                          }
  76:                                      }).ToArray()
  77:                          if (data && data.length > 0)
  78:                              deferred.resolve(data);
  79:                          else
  80:                              deferred.resolve([]);
  81:                      }
  82:                      else {
  83:                          deferred.resolve([]);
  84:                      }
  85:                  });
  86:              }
  87:   
  88:              return deferred.promise;
  89:   
  90:             
  91:          };
  92:   
  93:   
  94:          function provideAddress() {
  95:              if ($scope.addressCallback && angular.isFunction($scope.addressCallback)) {
  96:                 
  97:                  $scope.addressCallback({address:$scope.address});
  98:              }
  99:              else {
 100:                  if ($scope.address && $scope.address.city && $scope.address.state) {
 101:                      common.$broadcast(config.events.ADDRESS_COPIED, { address: $scope.address });
 102:                  }
 103:              }
 104:              
 105:          }
 106:   
 107:          function _extractState(comps) {
 108:              return Enumerable.From(comps)
 109:                          .Where(function (comp) {
 110:                              return Enumerable.From(comp.types)
 111:                                          .Any(function (type) {
 112:                                              return type.toLowerCase() == "administrative_area_level_1"
 113:                                          })
 114:                          })
 115:                          .Select(function (comp) {
 116:                              return {
 117:                                  name: comp.long_name,
 118:                                  code: comp.short_name
 119:                              }
 120:                          }).FirstOrDefault()
 121:          }
 122:          function _extractCity(comps) {
 123:              return Enumerable.From(comps)
 124:                          .Where(function (comp) {
 125:                              return Enumerable.From(comp.types)
 126:                                          .Any(function (type) {
 127:                                              return type.toLowerCase() == "neighborhood" || type.toLowerCase() == "locality";
 128:                                          })
 129:                          })
 130:                          .Select(function (comp) {
 131:                              return comp.long_name;
 132:                          }).FirstOrDefault()
 133:          }
 134:   
 135:          function _extractStreetAddress(comps) {
 136:              return Enumerable.From(comps)
 137:                          .Where(function (comp) {
 138:                              return Enumerable.From(comp.types)
 139:                                          .Any(function (type) {
 140:                                              return type.toLowerCase() == "route";
 141:                                          })
 142:                          })
 143:                          .Select(function (comp) {
 144:                              return comp.long_name;
 145:                          }).FirstOrDefault()
 146:          }
 147:          function _extractStreetNo(comps) {
 148:              return Enumerable.From(comps)
 149:                          .Where(function (comp) {
 150:                              return Enumerable.From(comp.types)
 151:                                          .Any(function (type) {
 152:                                              return type.toLowerCase() == "street_number";
 153:                                          })
 154:                          })
 155:                          .Select(function (comp) {
 156:                              return comp.long_name;
 157:                          }).FirstOrDefault()
 158:          }
 159:          function _extractCountry(comps) {
 160:              return Enumerable.From(comps)
 161:                          .Where(function (comp) {
 162:                              return Enumerable.From(comp.types)
 163:                                          .Any(function (type) {
 164:                                              return type.toLowerCase() == "country";
 165:                                          })
 166:                          })
 167:                          .Select(function (comp) {
 168:                              return {
 169:                                  name: comp.long_name,
 170:                                  code: comp.short_name
 171:                              }
 172:                          }).FirstOrDefault()
 173:          }
 174:          function _extractPostalCode(comps) {
 175:              return Enumerable.From(comps)
 176:                          .Where(function (comp) {
 177:                              return Enumerable.From(comp.types)
 178:                                          .Any(function (type) {
 179:                                              return type.toLowerCase() == "postal_code";
 180:                                          })
 181:                          })
 182:                          .Select(function (comp) {
 183:                              return comp.long_name;
 184:                          }).FirstOrDefault()
 185:          }
 186:      }
 187:   
 188:    
 189:      
 190:  })();



The template looks like this:


   1:  <label class="control-label block">
   2:      Address Lookup: 
   3:  </label>
   4:  <input type="text" ng-model="address" placeholder="Start typing full address here" typeahead-min-length="5" typeahead="address as address.title for address in getLocation($viewValue)" typeahead-loading="loadingLocations" class="control-l" typeahead-wait-ms="500" />
   5:  <i ng-show="loadingLocations" class="fa fa-refresh"></i>
   6:  <address ng-if="address.city">
   7:      <p>{{address.streetAddress}}</p>
   8:      <p>{{address.city}}, {{address.state.name}} {{address.postal_code}}</p>
   9:  </address>
  10:  <span class="btn btn-primary" ng-if="address.city" ng-click="provideAddress()">Copy Address</span>



When you use it in view, the definition would look like this:

<div st-postal-lookup address-callback="vm.copyAddress(address)"></div>

Basically, when address lookup’s copy address is clicked, vm.copyAddress will be called and then you can do whatever you like it.

Monday, December 7, 2015

Managing ALM in SharePoint 2010/2013 Projects using SPSD

Before I dive deeper into the using SPSD, let let me make a point that the concepts discussed here are primarily used on Server Side full trust code on SharePoint Platform. Theoretically, similar concepts can be applied to any Development Model but SPSD is not meant for that, as far as I can tell.

All seasoned SP Developers have one way or another to deploy custom WSP solutions. Very popular approach is to use a tool or script to deploy all WSPs put in a specific folder and he script/tool would uninstall existing WSPs or upgrade existing WSPs with new one. Initially, I downloaded SPSD for the same purpose but soon realized that deploying WSPs is just a fraction of functionality of SPSD and that it supports PowerShell based extensions which can be executed before and after WSP deployments. Basically, all my creative antennas got activated and I was immediately convinced that this thing is powerful!

In this article, I will throw the ideas that makes it a very appealing tool. This blog is not to learn to use SPSD, for that please refer to the Documentation tab of SPSD tool. Even after watching the presentation, it took be few more hours before I could fully grasp the functionality and start writing my own extension.

So at this point, I assume you are already familiar with SPSD or you have referred the documentation of it. I will begin with some useful tips and ideas here.

Idea # 1 – Automate .. Automate .. Automate

While I am a huge fan of automation, I don’t put extra ordinary efforts to achieve that. SPSD is a perfect balance and I think once you put efforts in SPSD automation, you can reuse them over and over again in other projects (again, don’t forget that we are talking about Server Side Development).

In one of my project, I used this tool towards the end of the project to deploy to QA/STAGE/PROD environments. For each target environment, I knew the server name so I created SERVERNAME.XML file under the SPSD Environments Directory and all environment specific variables were defined in the XML file.

The entire SPSD folder was added to source control and entire folder was given to Administrators and they were instructed to run deploy.bat and follow on-screen instructions.

WSPs must have been copied to Solutions directory, before the build was prepared. You can automate this using MSBuild if you want to get fancier but for my purpose, I write a simple .bat file and dropped it into SPSD’s solutions directory, and right before the build, I would run it to copy all my WSPs in that folder.

xcopy/y "..\..\lib\*.wsp" "."

I wrote quite a few custom extensions and was able to cut down initial build installation from 2.5 hours (of manual and automated combination) to 10 minutes (with nearly 99% automation via SPSD).

Idea # 2 – Use it from Day 1 in Development and integration

Building upon my previous experience, I thought of using SPSD from day – even in the development environment and no wonder, it was promising and successful experiment.

This time, on a separate project, I set it up such that a developer can get started with creating site collection, deploying wsps, activating features, creating term sets, building search schema and so on right from deploy.bat. So in this kind of setup, follow this guidelines:

1. Setup Regular SharePoint Projects (with WSP output) – Use this to deploy standard SharePoint artifacts such as Site Columns, Content Types, Page Layouts, Master Pages, Lists etc.

2. Add SPSD files to Source control and write scripts for things such as: create site collection (dev only), activate feature, create term sets, extend user profiles properties, search schema, configure permissions, Create SP Groups, create test or real content so on and so forth. Well, you got it – things that are one time in nature or things that can be overridden  by users should go here.

Now, working in a team always a challenge in custom SharePoint projects, so tool like SPSD can help a lot. Every day morning, or sometime several times in a day I would run deploy.bat and delete my site collection, create new one, deploy latest WSPs, activate features etc. – so I know that our code is 100% integrated – if there are any issues they will be caught as part of this process. I even created which creates publishing pages and provision web parts on the pages so I don’t have to worry about losing the momentum by recreating my development site collection every day.

Well, only this process gave me 100% confidence (I know this is a very string statement – but I really mean it) that when we are deployed to QA/PROD, things are **just** going to work. I don’t like surprises.

Tip – Create extensions.. and a lot of extensions

Writing an SPSD extension is one time investment but you can rip the benefits for many many months to come, I can guarantee that. SPSD comes with some extensions, test them well before using them. I had bad experience with an extension that creates crawled and managed properties. It did not work as I expected so I modified it – don’t hesitate to do that.

I may post several extensions that I wrote in a separate blog posts. But below are a few examples:

1. Create Term Groups and Term Sets

2. Search Schema

3. Create Publishing Pages and provision web parts

4. Activate Features

5. Create SP Groups and Put people in it

If have client/project specific things, you can just throw in a one time extensions with some hard coded values and scripts – like I say I like to strike a balance and not go too crazy if I don’t expect to reuse it. For instance, creating a project specific folder structure in document library or setting up versioning settings in project specific lists and libraries.

Tip – Ask Permissions

Before my extension is executed, my extension supports showing message to the user – describing what will happen next (any warning in yellow color) and ask the user to specify y/n – if answer is y only then the extension would execute. Of course, whether to prompt or not is also configurable in extension input. But this proved very handy during development when I can skip certain extensions that don’t truly belong to my own development effort. This is also handy for SP Administrators because they know what exactly will happen next when they say ‘y’ against the prompt. So installation becomes interactive and informative for administrators and developers.

Idea # 3 – think outside the box

I don’t believe SPSD provides a way to do versioning. So every time you run deploy.bat –everything specified in XML file is executed. Now this may not fit well into overall Application Life Cycle Management where v1 of application is deployed using SPSD but now you want to implement v1.1 of it. Well, with SPSD you can comment out the original extensions and just keep the v1.1 related extensions. I find this very inconvenient, ideally I would like it to be smart enough to bring me to v1.1 no matter if I start from beginning or from v1.0.

So what can we do about it? Feel free to leave your comments. I have some ideas about it but have not implemented it yet, if I do – expect another blog post for sure.

Tuesday, December 1, 2015

Strange/Unexpected/Frustrating in-place search Issue with SharePoint 2013

I am going to talk about poorly documented feature of SharePoint – the in-place search. Some of you may have already used it but you never realized that an official name for that is "in-place" search. Yes, I'm talking about that little search box appearing in document libraries and list, which allows you to search within current document library or list.

One of our clients reported a weird and very annoying issues while using that in-pace search. Examples below:

1. When I search for a word (say rate), it does not display documents which has "rate" as in title or name

2. When two different people who have been using the system for a while performs same keyword search they see different results even though they both have exact permissions

3. When I search for Title:rate, the documents containing word "rate" in title appears correctly. Then why not in case of #1 above?

4. When I search for word "rate" in regular search it shows the document containing the word risk in title. Then question is does in-place search use search engine internally?

5. Following message pops up randomly below the results:

Some files might be hidden. Include these in your search

When I fire up a query in regular search, the results are ranked and all documents are returned as expected – so there is absolutely no issue in crawling or querying components of search in general. The results of in-place search returned items for which the term was inside the content of files, so it's definitely using search component internally.

What's happening here? The issue was so embarrassing and annoying, for few hours I had absolutely no idea what was going on. We did not customize anything, so I believed strongly that there is some problem in SharePoint implementation or its by design. Regardless of the reason, it was unacceptable.

So I started digging deeper into it by opening SharePoint Assemblies in ILSpy and figured out following disturbing facts which are not so well explained anywhere!

1. That the in-place search would never return more than 500 results even though search would have returned more than 500.

2. It limits the results to current document library by appending "path:{{Root Folder Path of Document Lib/List}} AND {{query term}}"

3. It will ignore the result types and query rules.

4. While using search component, it looks for ID of the list items and path of the items

5. Another query is then generated to get all the fields that the Library/List view is expecting.  THIS IS THE WORST PART.  The first problem is the Performance. You clearly see that after search has returned items, SharePoint is still trying to query the list and get the corresponding list items. The worst of all is that during this process, it hits the 5000 threshold and it never returns the full results that search returned.

6 If that was not enough, the results are sorted based on view's sorting order, regardless of the search ranking. For an end user this may not be a good experience because the documents containing the search word may not appear on first few pages or may not appear at all if the document library has more than 5000 items.

So what are our options?

1. Ask client to increase threshold to match number of documents?

2. Ask clients to use more advanced search. Ask them to use FileExtension:docx or Title:Rate?

3. Suggest them not to use the in-place search?

4. Reorganize their libraries?

Should Microsoft feel proud about this in-place search?

I would love to find a working solution for this. Feel free to leave comments!

- See more at: https://bigapplesharepoint-public.sharepoint.com/pages/View-An-Insight.aspx?BlogID=120#sthash.3jjOreoa.dpuf

Saturday, November 21, 2015

Tiny but mighty Angular Directive: Record Status Indicator

In posts with similar title, I am planning to show some small Angular Directives that are not huge in terms of code but provide very useful functionality and greatly enhances the user experience.

You may have come across business applications in which records can be soft deleted – means they exists in the system but users are not allowed (or encouraged) to select those records. Such data preservation is helpful especially when historical data is required to be maintained in the system.

So I thought it would be nice to visually indicate if a particular record that user is looking at on screen is active or not. Consider the list of records below. The tiny status indicator just before the indicator clearly shows id record is active or inactive.

SNAGHTML378d23c1

Another example in the system, when user is trying to select a Carrier Authority in a dropdown:

SNAGHTML3793f07a

The dropdown includes all values but clearly shows that users should not use Inactive records. Like I said, the code for showing such status indicator is very tiny but brings a lot of value for the end users.
The code for the status indicator is provided below, you need to keep two things in mind:
1. It requires an entity with isActive property set to true or false depending in status of the record in the system
2. It uses Font Awesome icons and some color styles. You need to make sure Font Awesome and appropriate styles are included in your project.
app.directive('stStatusIndicator', function () {
return {
restrict: 'A',
scope: {
entity: '=stStatusIndicator'
},
link: function (scope, element, attr, ctrl) {
scope.$watch('entity', function (newValue, oldValue) {
if (newValue == undefined || newValue == null) {
element.empty();
}
})
scope.$watch('entity.isActive', function (newValue, oldValue) {
if (newValue == undefined)
return;

element.empty();
var node = document.createElement("i");
if (newValue == true) {
node.className = "fa fa-circle text-green hidden-print";
node.title = "Active";
}
else {
node.className = "fa fa-ban text-danger hidden-print";
node.title = "Inactive";
}
element.append(node);
})


}
};
});



Such visual indications can definitely be extended to other concepts such as indicating status of record in workflow, recently updated record so on and so forth. Hope you enjoyed the technique!

Thursday, November 19, 2015

Have TypeScript working with VS.NET 2013/2015 SharePoint Project Template

Assuming that…

1) You have latest TypeScript for Visual Studio installed

2) You have a SharePoint project and you add TypeScript file to it. Everything appears fine but you can’t generate .js file when you save it.

 

Following steps should solve the problem:

Step 1

Go to

C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\SharePointTools (For Visual Studio 2013, that is: C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v12.0\SharePointTools) and open Microsoft.VisualStudio.SharePoint.target file in notepad

Step 2

Add following just before the <Import Project="$(CustomBeforeSharePointTargets)" …… line in the targets file

<Import Project="$(MSBuildExtensionsPath32)\Microsoft\VisualStudio\v$(VisualStudioVersion)\TypeScript\Microsoft.TypeScript.targets" Condition="Exists('$(MSBuildExtensionsPath32)\Microsoft\VisualStudio\v$(VisualStudioVersion)\TypeScript\Microsoft.TypeScript.targets')" />

Step 3

Save the file and re-open VS.NET with SharePoint project. Try saving the .ts files and now you should have corresponding .js files generated.

Basically, the above target calls the Microsoft.TypeScript.targets which eventually will run the TypeScript compiler when .ts files are saved.

Step 4 (Optional)

Although the targets calls the right version of compiler, you might following problem:

After installing TypeScript 1.6 for VS.NET, typing “tsc –v” in package manager should result in 1.6.x. If not, it’s very likely that you had older version of VS.NET which created a PATH variable with old TypeScript location.

To fix it:

Open PATH variable value of the computer, remove old value of TypeScript location and replace with following:

C:\Program Files (x86)\Microsoft SDKs\TypeScript\1.6

Reopen VS.NET and you should be good!

Friday, March 27, 2015

Building a Flexible SMS Response System

Building SMS response system was my long pending dream! A few years back, I worked with SMS Service provider. Back then, the rates were high and the interfaces were not so easy.  But today, most SMS services offer cheap rates and very intuitive and easy to use, well documented APIs. Twillio is my choice for today. Let’s begin with a problem that’s being addressed here:

Problem:

My service needs to receive an SMS message from customer and process it and respond with an SMS message. The ask is very simple but if you can imagine, processing these SMS messages requires a flexible and configurable architecture, instead of a lots of if..else statements. I am not tying to do text analytics here, I expect my users to send specific keywords in SMS. For example, if they need help, they may send “help me”, “options”, or even “*”! If they need a list of nearby bars, we expect them to send “bars near me” or “nearby bars”….. you got the idea!

My Design Goals:

1. Ability to process incoming message and determine a response

2. The logic should be flexible and configurable. For instance, if I expect my users to send “help me” in order to receive a help message from us, it should be easy and configurable so that I can provided additional phrases that users can send without changing the code.

3. Break the problems into smaller chunks and make sure each piece of code has single responsibility

Possible Solutions

Given the problem and the design goals, I can think of two possible solutions.

Alternate Solution 1: Have separate classes created and each understands only a specific incoming message and processes it. Consider each class as a node which has ability to understand the request and provide response. If you like, consider it as an HTTP Module. Then build some sort of framework which passes the incoming message to each node in sequential fashion until it finds a node which can process the message.

Alternate Solution 2: Leverage a Service Bus architecture. A message can be put in a queue and then you can one subscriber which can process the message. And yes, the subscriber can pull the message from Queue and use the Alternate Solution 1 (above) to process the message. You may also have multiple subscribers and use the Topics mechanism of Azure Service Bus.

DISCLAIMER: If you are looking for a most reliable and scalable way to process the SMS messages, I recommend Alternate Solution 2 but for the low volume and fairly reliable solution, Solution 1 is not a bad choice either. While solution 1 is not the best, I’ll talk about that in this post, leaving solution 2 for another blog post. I also do not claim that these are the only two solutions, there could be many ways to solve such problems at hand. Sending and receiving SMS using Twilio is very simple. I’ll not talk about the mechanics of sending and receiving SMS messages because that depends from one provider to another.

So let’s take a look how does it (Alternate Solution 1) look conceptually:

image

And let’s zoon in a little bit into the SMS Processing Pipeline component:

image

The SMS Node processor in our case is a singleton class which initializes (and caches in its on state – it’s a Singleton, after all) the pipeline (series of processing nodes). It also has a method to receive a message and send response by routing the message through processing node.

A processing node implements a contract

   1:  interface ISmsProcessorNode
   2:      {
   3:          void ProcessMessage(SmsInfo message, SmsProcessingEventProperties eventProps);
   4:      }




The message represents everything that a typical incoming SMS message may include. For example, Twilio passes you following information along with some other information which we will not care:


   1:   public class SmsInfo
   2:      {
   3:          public string Message { get; set; }
   4:          public string ReceivedFrom { get; set; }
   5:          public string FromCity { get; set; }
   6:          public string FromCountry { get; set; }
   7:          public string FromState { get; set; }
   8:          public string FromZip { get; set; }
   9:      }




SmsProcessingEventProperties serves as a mechanism to receive configuration from the Node Processor and also as a mechanism to send response/status back to the node processor.


   1:  public enum SmsProcessingEventStatus
   2:      {
   3:          Processed = 0,
   4:          Skipped = 1, 
   5:          Warning =2,
   6:          Error = 3
   7:   
   8:      }
   9:      public class SmsProcessingEventProperties
  10:      {
  11:          public SmsProcessingEventStatus Status { get; set; }
  12:          public string ErrorMessage { get; set; }
  13:          public XElement Configuration { get; set; }
  14:          public string Response { get; set; }
  15:      }



Finally, here is the event processor looks like:


   1:   public class XmlNodePipelineProcessor
   2:      {
   3:          private static volatile XmlNodePipelineProcessor instance;
   4:          private static object syncRoot = new Object();
   5:   
   6:          List<SmsProcessingNode> _nodes = new List<SmsProcessingNode>();
   7:   
   8:          private XmlNodePipelineProcessor() { }
   9:   
  10:          public static XmlNodePipelineProcessor Instance
  11:          {
  12:              get
  13:              {
  14:                  
  15:                  if (instance == null)
  16:                  {
  17:                      lock (syncRoot)
  18:                      {
  19:                          var processorFile = HttpContext.Current.Server.MapPath("~/App_Data/SmsNodeProcessors.xml");
  20:                          if (instance == null)
  21:                          {
  22:                              instance = new XmlNodePipelineProcessor();
  23:                              instance.InitializeFromXml(processorFile);
  24:   
  25:                          }
  26:   
  27:                      }
  28:                  }
  29:   
  30:                  return instance;
  31:   
  32:              }
  33:                  
  34:              
  35:          }
  36:   
  37:          #region Public Members
  38:          public IEnumerable<SmsProcessingNode> GetProcessingNodes()
  39:          {
  40:              return _nodes;
  41:          }
  42:          public string GetResponseFor(SmsInfo incomingMessage)
  43:          {
  44:              string response = string.Empty;
  45:   
  46:              foreach (SmsProcessingNode node in _nodes)
  47:              {
  48:                  try
  49:                  {
  50:                      Assembly assembly = Assembly.Load(node.Assembly);
  51:                      Object classInstance = assembly.CreateInstance(node.ClassName);
  52:                      ISmsProcessorNode processingNode = classInstance as ISmsProcessorNode;
  53:                      if (processingNode == null)
  54:                      {
  55:                          Trace.WriteLine(string.Format("Assembly:{0}, Class{1} is not an implementation of {2}.  Node-{3} will be ignored.", assembly, node.ClassName, typeof(ISmsProcessorNode).FullName, node.Name));
  56:                          continue;
  57:                      }
  58:                      SmsProcessingEventProperties eventProps = new SmsProcessingEventProperties()
  59:                      {
  60:                          Configuration = node.Configuration,
  61:                          ErrorMessage = string.Empty,
  62:                          Status = SmsProcessingEventStatus.Skipped
  63:                      };
  64:                      processingNode.ProcessMessage(incomingMessage, eventProps);
  65:   
  66:                      // Check the modified eventProps and take actions
  67:                      if (eventProps.Status == SmsProcessingEventStatus.Skipped)
  68:                          continue;
  69:   
  70:                      if (eventProps.Status == SmsProcessingEventStatus.Error)
  71:                      {
  72:                          Trace.WriteLine(string.Format("Incoming Msg:{0}, Node:{1}, Error:{2}", incomingMessage.Message, node.Name, eventProps.ErrorMessage));
  73:                          continue;
  74:                      }
  75:                      if (eventProps.Status == SmsProcessingEventStatus.Processed && node.ContinueAfterProcessing)
  76:                      {
  77:                          continue;
  78:                      }
  79:                      if (eventProps.Status == SmsProcessingEventStatus.Processed && !string.IsNullOrEmpty(eventProps.Response))
  80:                      {
  81:                          return eventProps.Response;
  82:                      }
  83:                      
  84:                  }
  85:                  catch (Exception ex)
  86:                  {
  87:   
  88:                      Trace.WriteLine(string.Format( "Node:{0} failed to process message:'{1}', error message: {2}", node.Name, incomingMessage.Message, ex.Message));
  89:                  }
  90:                  
  91:              }
  92:   
  93:              return string.Format( "Sorry, we could not process your message: {0}", incomingMessage.Message);
  94:          }
  95:          #endregion
  96:   
  97:          private void InitializeFromXml(string fileName)
  98:          {
  99:   
 100:   
 101:              XmlDocument doc = new XmlDocument();
 102:              try
 103:              {
 104:                  doc.Load(fileName);
 105:              }
 106:              catch (XmlException ex)
 107:              {
 108:                  Trace.Write(ex);
 109:   
 110:              }
 111:   
 112:              if (doc.DocumentElement.LocalName != "NodeProcessors")
 113:              {
 114:                  Trace.WriteLine("File does not contain NodeProcessors");
 115:                  return;
 116:              }
 117:   
 118:   
 119:              XmlNodeReader nodeReader = new XmlNodeReader(doc.DocumentElement);
 120:              XDocument xDoc = XDocument.Load(nodeReader);
 121:   
 122:              Dictionary<string, string> assemblyRefs = new Dictionary<string, string>();
 123:              foreach (var item in xDoc.Descendants("Assembly"))
 124:              {
 125:                  //TODO: validate that item has Name and Value attributes
 126:                  assemblyRefs.Add(item.Attribute("Name").Value, item.Attribute("Value").Value);
 127:              }
 128:   
 129:              foreach (var item in xDoc.Descendants("Node"))
 130:              {
 131:   
 132:   
 133:                  if (!item.HasAttributes
 134:                      || item.Attribute("Name") == null
 135:                      || string.IsNullOrEmpty(item.Attribute("Name").Value)
 136:                      || item.Attribute("AssemblyRef") == null
 137:                      || string.IsNullOrEmpty(item.Attribute("AssemblyRef").Value)
 138:                      || !assemblyRefs.ContainsKey(item.Attribute("AssemblyRef").Value)
 139:                      || item.Attribute("Class") == null
 140:                      || string.IsNullOrEmpty(item.Attribute("Class").Value))
 141:                  {
 142:   
 143:   
 144:                      continue;
 145:                  }
 146:                  // All data seems to be okay, process it
 147:                  SmsProcessingNode newNode = new SmsProcessingNode()
 148:                  {
 149:                      Assembly = assemblyRefs[item.Attribute("AssemblyRef").Value.Trim()],
 150:                      ClassName = item.Attribute("Class").Value.Trim(),
 151:                      Name = item.Attribute("Name").Value,
 152:                      StopOnError = item.GetAttributeAsBool("StopOnError", false),
 153:                      ContinueAfterProcessing = item.GetAttributeAsBool("ContinueAfterProcessing", false),
 154:                      Configuration = item.Element("Configuration") 
 155:                  };
 156:   
 157:                  _nodes.Add(newNode);
 158:              }
 159:   
 160:   
 161:             
 162:          }
 163:      }




A node representation is here:


   1:  public class SmsProcessingNode
   2:      {
   3:          public string Name { get; set; }
   4:          public string Assembly { get; set; }
   5:          public string ClassName { get; set; }
   6:   
   7:          private bool _continueAfterProcessing = false;
   8:   
   9:          public bool ContinueAfterProcessing
  10:          {
  11:              get { return _continueAfterProcessing; }
  12:              set { _continueAfterProcessing = value; }
  13:          }
  14:   
  15:          private bool _stopOnError;
  16:   
  17:          public bool StopOnError
  18:          {
  19:              get { return _stopOnError; }
  20:              set { _stopOnError = value; }
  21:          }
  22:   
  23:          public XElement Configuration { get; set; }
  24:         
  25:      }



The XML based Node Configuration will look something like this:


<?xml version="1.0" encoding="utf-8" ?>
<NodeProcessors>
  <References>
    <Assemblies>
      <Assembly Name="PhoneService" Value="FULL NAME OF YOUR ASSEMBLY"></Assembly>
    </Assemblies>
  </References>
    <Nodes>
      <Node Name="PersistMessageProcessor" AssemblyRef="PhoneService" Class="FULL NAME OF CLASS TO SAVE MESSAGE IN DATABASE" ContinueAfterProcessing="true" StopOnError="false">
        <Configuration>
 
        </Configuration>
      </Node>
      <Node Name="HelpTextProcessor" AssemblyRef="PhoneService" Class="FULL NAME OF A CLASS TO SEND A RESPONSE WHEN  INCOMING MESSAGE IS TO SEEK HELP" ContinueAfterProcessing="false" StopOnError="false">
        <Configuration>
          <CatchPhrases>
            <Phrase>help</Phrase>
            <Phrase>help me</Phrase>
            <Phrase>help please</Phrase>
            <Phrase>what are my options?</Phrase>
            <Phrase>options</Phrase>
          </CatchPhrases>
          <Response>
            YOU NEED HELP! HERE ARE YOUR OPTIONS:...
          </Response>
        </Configuration>
      </Node>
      
    </Nodes>
  
</NodeProcessors>




As you can see, there are two examples that I have configured. The intention of the first node is to save the message in the database. The intention of 2nd node is to figure out if user is looking for help and if so, send a help response which is also configurable. Note that Configuration element in above XML has particular schema, the framework (node processor) will pass the entire configuration and it will be up to the node implementation to figure out how to use it.


So how does my HelpTextProcessor look like? well, i think that’s a good example which complete the loop in understanding how exactly it works. And here it is:


 public void ProcessMessage(SmsInfo message, Models.SmsProcessingEventProperties eventProps)
        {
            
            if (eventProps.Configuration == null)
            {
                eventProps.Status = Models.SmsProcessingEventStatus.Skipped;
            }
            try
            {
                foreach (var phrase in eventProps.Configuration.Descendants("Phrase"))
                {
                    if (phrase.Value.ToLower() == message.Message.ToLower().Trim())
                    {
                        eventProps.Status = Models.SmsProcessingEventStatus.Processed;
                        eventProps.Response = eventProps.Configuration.Descendants("Response").FirstOrDefault().Value.Trim();
                    }
                }
            }
            catch (Exception ex)
            {
                eventProps.Status = Models.SmsProcessingEventStatus.Error;
                eventProps.ErrorMessage = ex.Message;
                
            }