Menu
Home / Blog Posts /

All posts in Troubleshooting

June 30, 2017
by

SharePoint Custom Solution Crashes IIS Worker Process (w3wp.exe) – Part 1

administration_slider
PowerShell, SharePoint, SharePoint 2013, Troubleshooting , , , , 0

Had a doozy of an issue the other day.  All of a sudden, a SharePoint farm that has been chugging along with no changes suddenly started having some weird issues.  Users could open, view, edit documents, but as soon as they attempted a save or an upload of a new document things started to go bad.  If they were using Windows Explorer they received the error: “The specified network name is no longer available”

SharePoint Custom Solution Crashes IIS Worker Proces - Windows Explorer Error

If they were using the GUI the upload form hung for a while and eventually reverted to “The Page Cannot be Displayed”

At the same time, we were getting reports of users in other areas of the farm getting a very slow response within SharePoint.  What was really confusing about this was that the issue was happening to just a single site collection in the farm.

Errors Received

Windows Event Log

We were receiving a number of errors besides those at the end user level.  The server event log indicated our app pool was crashing.  The error received was actually a warning (to me if an app pool is crashing, it should be an error) with the msg:

“A process serving application pool ‘SharePoint Web Apps’ suffered a fatal communication error with the Windows Process Activation Service. The process id was ‘6292’. The data field contains the error number.”

SharePoint Custom Solution Crashes IIS Worker Proces - EventLogError

In the multiple WFE environment it was happening back and forth between the two serves indicating the load balance was doing its job.  It also indicated why people were seeing slow response.  Each time the app pool failed, it had to restart and then reload the SharePoint environment (like you see after an IIS Reset).

ULS Logs

The ULS logs were something else.  In this particular environment our logs usually range from 5MB-40MB in size for a 30 min period.  When I ran a one minute log export using “Merge-SPLogFIle” the exported file was 1.3 GB.  Nothing screamed error at me, however there were a couple of things standing out.

06/21/2017 10:30:38.00        w3wp.exe (0x112C)        0x1E58        SharePoint Foundation        Performance        naqx        Monitorable        Potentially excessive number of SPRequest objects (16) currently unreleased on thread 46.  Ensure that this object or its parent (such as an SPWeb or SPSite) is being properly disposed. This object is holding on to a separate native heap. Allocation Id for this object: {C3DC973B-90B4-4974-A33D-A5A05A722DF7} Stack trace of current allocation:    at Microsoft.SharePoint.SPGlobal.CreateSPRequestAndSetIdentity(SPSite site, String name, Boolean bNotGlobalAdminCode, String strUrl, Boolean bNotAddToContext, Byte[] UserToken, SPAppPrincipalToken appPrincipalToken, String userName, Boolean bIgnoreTokenTimeout, Boolean bAsAnonymous)     at Microsoft.SharePoint.SPWeb.InitializeSPRequest()     at Microsoft.SharePoint.SPWeb.EnsureSPRequest()     at Microsoft.SharePoint.SPSite.OpenWeb(String strUrl, Int32 mondoHint)     at Microsoft.SharePoint.SPSite.OpenWeb(Guid gWebId, Int32 mondoHint)     at Microsoft.SharePoint.SPSite.OpenWeb(Guid gWebId)….

06/21/2017 10:31:21.09        w3wp.exe (0x112C)        0x1E58        SharePoint Foundation        General        8m90        Medium        1045 heaps created, above warning threshold of 128. Check for excessive SPWeb or SPSite usage.        a8dafd9d-9faa-70d5-b0e7-8c1711386713

So this screamed of some custom code (which we do have) running that is not disposing of the SPSite or SPWeb objects properly.  Why it suddenly became a problem I don’t know.  We did have security patches happen on the server over the weekend.  I didn’t think it likely to be the cause as the environment had been used for a day and a half with no issues.  We backed out of the patch anyways, but didn’t affect the issue occurring.  What was also confusing was this issue was also occurring in our Pre-Prod environment.  The silver lining is now I could really do some troubleshooting without affecting sites that were functioning or production data.

I finally tracked down the issue to an event receiver we have running in our environment.  The project sites all of the same structure and it was decided that code would be used to enforce this structure.  To that end, event receivers were built to ensure folders at certain levels (library root, root +1 level and root +2 levels) were not deleted nor files or folders at those levels were added.  I took a guess that these event receivers were causing the issues.  Using PowerShell I removed the event receivers from a library being affected.  In case you need this for something else the code to remove a list event receiver is:

$siteColURL = "<Site URL>";

$spSite = Get-SPSite $siteColURL

foreach($spWeb in $spSite.AllWebs)
{
    $spList = $spWeb.Lists["<LIBRARY NAME"];
    
    try 
    {
        for($i = $splist.EventReceivers.Count - 1; $i -ge 0; $i--)
        {
            $eventReceiver = $spList.EventReceivers[$i];
            if($eventReceiver.Class -eq "<EVENT RECEIVER CLASS>")
            {
                $eventReceiver.Delete();
            }
        }

        $spList.Update();
        
        Write-Host("Updated library for site: {0}" -f $spWeb.URL);
    }
    catch 
    {
        $ErrorMessage = $_.Exception.Message;
        Write-Host ("{0}: An error occurred deleting the event receiver for site: {1}.  Error received: {2}" -f `
                        (get-date).ToString("yyyy-MM-dd HH:mm:ss"), $spWeb.URL, $ErrorMessage) -ForegroundColor Red
    }    
}

In the above code (which removes the event receivers from ALL specified libraies in ALL subsites) I used the event receiver class to find the items I wanted to remove.  You can also use .Name and .Assembly if you wish. I used Class simply because when the sites were created and the receivers attached, no names were given.  With the event receivers removed, users were now able to upload and save documents.  So I had indeed found the culprit.  Now to determine why.

I’ll cover the review of the code and the final determination of the cause of the issue in Part 2 of this series.

 

Thanks for reading!

February 15, 2017
by

Field or Property ‘TimeZoneId’ does not exist when using SharePoint Search

Configuration
SharePoint 2013, Troubleshooting , , 0

Our test environment was giving me issues today.  It’s an error that was reported and supposedly fixed nearly 18 months ago.  When typing a query in the text box, the initial search works, but when using a refiner or changing the query the result would return an error Field or Property ‘TimeZoneId’ does not exist.  This was first reported after the July 2015 CU as a regression and supposedly fixed in the August 2015 CU.  The problem was in the Microsoft.Office.Server.Search.ServerProxy.dll.  Now I admit our servers are out of date when it comes to CUs (something I have been trying to get time from my client to do) but we are well beyond August 2015 CU.

Our environment is setup with Search being provided by our App servers in a Shared Services configuration.  When checking the WFE I found the dll version to be at 15.0.4815.1000.  We have two app servers that are running the search services.  App#1 had the same version of the dll, but App#2 was actually sitting at an older version: 15.0.44020.1017.  I believed this to be the culprit.

Resolving: “Field or Property ‘TimeZoneId’ does not exist“

  1. Shut down IIS on the server you need to update.
  2. From a server with the correct version copy Microsoft.Office.Server.Search.ServerProxy.dll to the _app_bin folder of the SharePoint virtual directory (example: C:\inetpub\wwwroot\wss\VirtualDirectories\AppServer\_app_bin)
    • If you don’t have a “good” location, the correct file should also be located in “C:Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\CONFIG\BIN”
  3. Restart IIS and test.  Should be fine now.

The cause is likely this single file not being copied over properly when patched.  Hope this helps someone still having this issue.

 

Thanks for reading!

January 4, 2017
by

Unable to Add Existing Site Columns to Content Types in SharePoint

Configuration
SharePoint, SharePoint 2013, SharePoint 2016, Troubleshooting, Troubleshooting (SP 2016) 3

While deploying a new solution to our test environment for my client the other day I found that I was unable to add existing site columns to content types within the SharePoint site I was working in.  No real reason given.  As you can see from the screenshot below the GUI wasn’t much help:

Unable to Add Existing Site Columns to Content Type - GUI Error

However, as always SharePoint provided the handy-dandy correlation id.  Using that I was able to get a great deal more information on the problem.

01/03/2017 13:18:07.08 	w3wp.exe (ServerName:0x087C)         	0x2D4C	SharePoint Foundation         	General                       	8nca	Medium	Application error when access /_layouts/15/fldpick.aspx, Error=No two choices should have the same ID   at Microsoft.SharePoint.ApplicationPages.ChoiceComparerWithDefaultGroup.Compare(PickerItem c1, PickerItem c2)     at System.Collections.Generic.ArraySortHelper`1.DepthLimitedQuickSort(T[] keys, Int32 left, Int32 right, IComparer`1 comparer, Int32 depthLimit)     at System.Collections.Generic.ArraySortHelper`1.DepthLimitedQuickSort(T[] keys, Int32 left, Int32 right, IComparer`1 comparer, Int32 depthLimit)     at System.Collections.Generic.ArraySortHelper`1.DepthLimitedQuickSort(T[] keys, Int32 left, Int32 right, IComparer`1 comparer, Int32 depthLimit)     at System.Collections.Generic.ArraySortHelper`1.DepthLimitedQuickSort(T[] keys, Int32 left, Int32 right, IComparer`1 comparer, Int32 de...	137fc79d-a097-403e-4c78-ff67da2e62e2
01/03/2017 13:18:07.08*	w3wp.exe (ServerName:0x087C)         	0x2D4C	SharePoint Foundation         	General                       	8nca	Medium	...pthLimit)     at System.Collections.Generic.ArraySortHelper`1.Sort(T[] keys, Int32 index, Int32 length, IComparer`1 comparer)	137fc79d-a097-403e-4c78-ff67da2e62e2
01/03/2017 13:18:07.08 	w3wp.exe (ServerName:0x087C)         	0x2D4C	SharePoint Foundation         	Runtime                       	tkau	Unexpected	System.ArgumentException: No two choices should have the same ID    at Microsoft.SharePoint.ApplicationPages.ChoiceComparerWithDefaultGroup.Compare(PickerItem c1, PickerItem c2)     at System.Collections.Generic.ArraySortHelper`1.DepthLimitedQuickSort(T[] keys, Int32 left, Int32 right, IComparer`1 comparer, Int32 depthLimit)     at System.Collections.Generic.ArraySortHelper`1.DepthLimitedQuickSort(T[] keys, Int32 left, Int32 right, IComparer`1 comparer, Int32 depthLimit)     at System.Collections.Generic.ArraySortHelper`1.DepthLimitedQuickSort(T[] keys, Int32 left, Int32 right, IComparer`1 comparer, Int32 depthLimit)     at System.Collections.Generic.ArraySortHelper`1.DepthLimitedQuickSort(T[] keys, Int32 left, Int32 right, IComparer`1 comparer, Int32 depthLimit)     at System.Collections....	137fc79d-a097-403e-4c78-ff67da2e62e2
01/03/2017 13:18:07.08*	w3wp.exe (ServerName:0x087C)         	0x2D4C	SharePoint Foundation         	Runtime                       	tkau	Unexpected	...Generic.ArraySortHelper`1.Sort(T[] keys, Int32 index, Int32 length, IComparer`1 comparer)	137fc79d-a097-403e-4c78-ff67da2e62e2
01/03/2017 13:18:07.08 	w3wp.exe (ServerName:0x087C)         	0x2D4C	SharePoint Foundation         	General                       	ajlz0	High	Getting Error Message for Exception System.Web.HttpUnhandledException (0x80004005): Exception of type 'System.Web.HttpUnhandledException' was thrown. ---> System.InvalidOperationException: Failed to compare two elements in the array. ---> System.ArgumentException: No two choices should have the same ID     at Microsoft.SharePoint.ApplicationPages.ChoiceComparerWithDefaultGroup.Compare(PickerItem c1, PickerItem c2)     at System.Collections.Generic.ArraySortHelper`1.DepthLimitedQuickSort(T[] keys, Int32 left, Int32 right, IComparer`1 comparer, Int32 depthLimit)     at System.Collections.Generic.ArraySortHelper`1.DepthLimitedQuickSort(T[] keys, Int32 left, Int32 right, IComparer`1 comparer, Int32 depthLimit)     at System.Collections.Generic.ArraySortHelper`1.DepthLimitedQuickSort(T[] keys, ...	137fc79d-a097-403e-4c78-ff67da2e62e2
01/03/2017 13:18:07.08*	w3wp.exe (ServerName:0x087C)         	0x2D4C	SharePoint Foundation         	General                       	ajlz0	High	...Int32 left, Int32 right, IComparer`1 comparer, Int32 depthLimit)     at System.Collections.Generic.ArraySortHelper`1.DepthLimitedQuickSort(T[] keys, Int32 left, Int32 right, IComparer`1 comparer, Int32 depthLimit)     at System.Collections.Generic.ArraySortHelper`1.Sort(T[] keys, Int32 index, Int32 length, IComparer`1 comparer)     --- End of inner exception stack trace ---     at System.Collections.Generic.ArraySortHelper`1.Sort(T[] keys, Int32 index, Int32 length, IComparer`1 comparer)     at System.Collections.Generic.List`1.Sort(Int32 index, Int32 count, IComparer`1 comparer)     at Microsoft.SharePoint.WebControls.GroupedItemPicker.GetDataString(StringBuilder sb)     at Microsoft.SharePoint.WebControls.GroupedItemPicker.Render(HtmlTextWriter writer)     at System.Web.UI.Control.Render...	137fc79d-a097-403e-4c78-ff67da2e62e2
01/03/2017 13:18:07.08*	w3wp.exe (ServerName:0x087C)         	0x2D4C	SharePoint Foundation         	General                       	ajlz0	High	...ControlInternal(HtmlTextWriter writer, ControlAdapter adapter)     at ASP._layouts_15_fldpick_aspx.__Render__control16(HtmlTextWriter __w, Control parameterContainer)     at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children)     at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)     at ASP._controltemplates_15_inputformsection_ascx.__Render__control1(HtmlTextWriter __w, Control parameterContainer)     at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children)     at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)     at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children)     at System.Web.UI.Cont...	137fc79d-a097-403e-4c78-ff67da2e62e2
01/03/2017 13:18:07.08*	w3wp.exe (ServerName:0x087C)         	0x2D4C	SharePoint Foundation         	General                       	ajlz0	High	...rol.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)     at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children)     at Microsoft.SharePoint.WebControls.AjaxDelta.RenderChildren(HtmlTextWriter output)     at System.Web.UI.WebControls.WebControl.RenderContents(HtmlTextWriter writer)     at System.Web.UI.WebControls.WebControl.Render(HtmlTextWriter writer)     at Microsoft.SharePoint.WebControls.AjaxDelta.Render(HtmlTextWriter writer)     at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)     at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children)     at System.Web.UI.HtmlControls.HtmlForm.RenderChildren(HtmlTextWriter writer)     at System.Web.UI.HtmlContr...	137fc79d-a097-403e-4c78-ff67da2e62e2
01/03/2017 13:18:07.08*	w3wp.exe (ServerName:0x087C)         	0x2D4C	SharePoint Foundation         	General                       	ajlz0	High	...ols.HtmlContainerControl.Render(HtmlTextWriter writer)     at Microsoft.SharePoint.WebControls.SharePointForm.Render(HtmlTextWriter output)     at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)     at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children)     at System.Web.UI.HtmlControls.HtmlContainerControl.Render(HtmlTextWriter writer)     at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)     at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children)     at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)     at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection ...	137fc79d-a097-403e-4c78-ff67da2e62e2
01/03/2017 13:18:07.08*	w3wp.exe (ServerName:0x087C)         	0x2D4C	SharePoint Foundation         	General                       	ajlz0	High	...children)     at Microsoft.SharePoint.WebControls.UnsecuredLayoutsPageBase.RenderChildren(HtmlTextWriter writer)     at System.Web.UI.Page.Render(HtmlTextWriter writer)     at Microsoft.SharePoint.WebControls.DeltaPage.RenderToBase(HtmlTextWriter writer)     at Microsoft.SharePoint.WebControls.DeltaPage.Render(HtmlTextWriter writer)     at Microsoft.SharePoint.WebControls.UnsecuredLayoutsPageBase.Render(HtmlTextWriter writer)     at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)     at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)     at System.Web.UI.Page.HandleError(Exception e)     at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean include...	137fc79d-a097-403e-4c78-ff67da2e62e2
01/03/2017 13:18:07.08*	w3wp.exe (ServerName:0x087C)         	0x2D4C	SharePoint Foundation         	General                       	ajlz0	High	...StagesAfterAsyncPoint)     at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)     at System.Web.UI.Page.ProcessRequest()     at System.Web.UI.Page.ProcessRequest(HttpContext context)     at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()     at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)	137fc79d-a097-403e-4c78-ff67da2e62e2

Specifically, the problem was: “No two choices should have the same ID“.  Looking at the stack trace you can see that it looks like SharePoint is looking at the fields that exist in the site.  This makes sense because it has to build that list for you to choose from doesn’t it?  I scanned through my list of site columns and noticed that for some reason I had two columns called “Hashtags”.  I am not entirely sure what this field is for, but I believe it is added with the Newsfeed.

Unable to Add Existing Site Columns to Content Type - Multiple Hashtags

So the next step?  Well let’s delete the extra column.  Attempting to delete via the GUI just resulted in a window that never provided a response, nor deleted the offending field.  Next onto PowerShell.  While getting the field object I found that the two fields actually existed with the exact same GUID.  So that’s where the error message above came from.  While deleting via PowerShell I got a bit further and received the message: “Site columns which are included in content types cannot be deleted“.   I didn’t have a content type sitting there with Hashtags within it, so ran a script to go through each content type and to look for the field.

There were two hidden, system content types Project Policy and System Media Collection that contained the site column.  I did a few things to try and remove the site column from the content types, but I am not going to outline here because I don’t want to give any of you ideas (they aren’t something you want to do in a production environment and they didn’t work anyways).  Needless to say, I did everything I could think of but I could not remove the extra field from the sites.

Placed a call with Microsoft to see if they had any suggestions.  Apparently, according to the support engineer I worked with, Microsoft has seen this before.  Nowhere in the vast interwebs did I find this information.  One reason for this post is for prosperity ;-).  The fix is actually really simple.

Removing the Extra Field

Microsoft provided me with the feature definition of the Hashtags field.  The solution was to install the feature into the farm, activate the feature on the sub sites that had the Hashtags field existing within (whether they had the double instance or not) and then disabling the feature within the site.  Finally, we removed the feature from the farm as well.  I will provide that feature definition in this post, but want to make something very clear.  I did not create this, nor do I provide any warranty or take any responsibility if applying the solution causes instability in your farm.

You can download the feature here.

To apply the feature to correct your farm, follow these steps.

  1. Extract the zip file and place the MMSField folder in the directory: C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\TEMPLATE\FEATURES.
  2. Open a SharePoint PowerShell window and run the following command:
Install-spfeature -path MMSField
  1. After the feature is added to the farm run the following script to check each sub site within your site collection to add and remove the feature (this step will force the system to remove the field and re-add it properly).
$spSite = Get-SPSite <URL>

foreach ( $spWeb in $spSite.AllWebs)
{
    $spField = $spWeb.Fields['HashTags']
    if ( $spField -ne $null )
    {
        Write-host $spWeb.url, $spField.Title
        Enable-SPFeature -Identity MMSField -Url $spWeb.url -Force -Confirm:$false
        Disable-SPFeature -Identity MMSField -Url $spWeb.url -Force -Confirm:$false
    }
}
  1. Remove the solution from the farm by running the following command:
Uninstall-spfeature -identity MMSField

Once this is complete you will have removed the extra instances of the HashTags field and can now add site columns to your content types without issues.

 

Thanks for reading!

November 28, 2016
by

Accessing SharePoint Site: “The context has expired and can no longer be used”

Configuration
SharePoint, SharePoint 2013, Troubleshooting , , 0

Just a quick little note of a quick solution I came up for a weird problem that occurred on my dev server.  I was preparing for a presentation and when accessing the site that contained all settings, data and code for the presentation was immediately receiving the error message: “The context has expired and can no longer be used”.

The context has expired and can no longer be used - Error Message

Read more

September 27, 2016
by

Tip for Backing Up with Export-SPWeb and Large Data Sets

administration_slider
Configuration (SP 2016), SharePoint, SharePoint 2013, SharePoint 2016, Troubleshooting, Troubleshooting (SP 2016) , , , 0

Today I was in the middle of testing how our environment works with large content databases and ran into an issue with the command Export-SPWeb and large data sets.  This might be known already do some of you, but I have been working with SharePoint a long time and didn’t know this, so wanted to share in case others ran into the same or similar problem.

Read more

Older Entries
Newer Entries