The object was not found. (Error from SharePoint site: HttpStatusCode NotFound The request failed with HTTP status 404: Not Found.)

I was checking the crawl logs for my Search Service application and was seeing the below error many times

The object was not found. (Error from SharePoint site: HttpStatusCode NotFound The request failed with HTTP status 404: Not Found.)

Cause

This because a file that was previously stored in Sharepoint and the crawler couldn’t find the object anymore. This could have been because the file was deleted or the access was denied. So the crawler would keep on trying to crawl the object for 30 times or 720 hours.

When the crawler encounters an access denied or a file not found error, the item is deleted from the index if the error was encountered in more than ErrorDeleteCountAllowed (the value for this is 30) consecutive crawls AND the duration since the first error is greater than ErrorDeleteIntervalAllowed (the value for this is 720 hours or 30 days). If both conditions are not met, the item is retried.

We can manage this deletion policy at the service application level to specify how many times and how long an item will be retried in consecutive crawls. This policy existed in Microsoft Office SharePoint Server 2007, but you could not view them nor change their values.

Resolution

Either you wait for 30 days for the errors to go away or you modify the default values.

$ssa = Get-SPEnterpriseSearchServiceApplication

To view the current value of a property:

$ssa.GetProperty("<PropertyName>")

To change the value of a property:

$ssa.SetProperty("<PropertyName>", <NewValue>)

The Property Name here would be ErrorDeleteCountAllowed and ErrorDeleteIntervalAllowed 

Read this article in TechNet to know more about the other deletion policies in SharePoint 2010 Search