Jump to content
SGauvin

URGENT - upgrading from 7.6 failed

Recommended Posts

HI,

 

I'm attempting to upgrade to version 8 from 7.6 and I got the following:

 

2018-11-26 10:20:12 PM - Preparing for upgrade has completed successfully.
2018-11-26 10:38:36 PM - Build 7668 Upgrade Started.
2018-11-26 10:38:36 PM - Build 7668- Step 1 Completed Successfully.
2018-11-26 10:38:36 PM - Build 7668 - Step 2 Completed Successfully.
2018-11-26 10:38:36 PM - Build 7668 - Step 3 Completed Successfully.
2018-11-26 10:38:36 PM - Build 7668 completed successfully.
2018-11-26 10:38:37 PM - Build 7676 Upgrade Started.
2018-11-26 10:38:37 PM - Build 7676 - Step 1 Completed Successfully.
2018-11-26 10:38:37 PM - Build 7676 completed successfully.
2018-11-26 10:38:38 PM - Build 7721 Upgrade Started.
2018-11-26 10:38:38 PM - Build 7721- Step 1 Completed Successfully.
2018-11-26 10:38:39 PM - Build 7721 - Step 2 Completed Successfully.
2018-11-26 10:38:47 PM - Build Process 'Build_7721_CopyPassword()' failed with the following error - Thread+was+being+aborted. ::: +++at+System.Threading.Thread.AbortInternal()%0d%0a+++at+System.Threading.Thread.Abort(Object+stateInfo)%0d%0a+++at+System.Web.HttpResponse.AbortCurrentThread()%0d%0a+++at+Passwordstate.Common.AddDebugInfoData(String+Category%2c+String+DebugInformation%2c+String+EventType)%0d%0a+++at+Passwordstate.Audit.AddAuditing(String+strActivity%2c+String+strDescription%2c+Int32+intPasswordListID%2c+Int32+intPasswordID)%0d%0a+++at+Passwordstate.Upgrade.Build_7721_CopyPassword(String+PasswordID%2c+String+HostID%2c+String+ScriptID%2c+Boolean+ResetStatus%2c+DateTime+LastResetDate%2c+Boolean+AuditRecordFound%2c+String+HostName%2c+String+UserName)


2018-11-26 10:38:47 PM - Build Process 'Build_7721_UpdateNonActiveDirectoryAccounts()' failed with the following error - Thread+was+being+aborted. ::: +++at+System.Threading.Thread.AbortInternal()%0d%0a+++at+System.Threading.Thread.Abort(Object+stateInfo)%0d%0a+++at+System.Web.HttpResponse.AbortCurrentThread()%0d%0a+++at+Passwordstate.Upgrade.Build_7721_CopyPassword(String+PasswordID%2c+String+HostID%2c+String+ScriptID%2c+Boolean+ResetStatus%2c+DateTime+LastResetDate%2c+Boolean+AuditRecordFound%2c+String+HostName%2c+String+UserName)%0d%0a+++at+Passwordstate.Upgrade.Build_7721_UpdateNonActiveDirectoryAccounts()


2018-11-26 10:38:47 PM - Build Process 'Build_7721_Updates' failed with the following error - Thread+was+being+aborted. ::: +++at+System.Threading.Thread.AbortInternal()%0d%0a+++at+System.Threading.Thread.Abort(Object+stateInfo)%0d%0a+++at+System.Web.HttpResponse.AbortCurrentThread()%0d%0a+++at+Passwordstate.Upgrade.Build_7721_UpdateNonActiveDirectoryAccounts()%0d%0a+++at+Passwordstate.Upgrade.Build_7721_Updates()

 

 

Hoping you can help immediately, before I have to back out.  Is there an easy fix/quick fix for this?

 

Thanks. ... Scot

Share this post


Link to post
Share on other sites

Hi Scot,

 

We don't think there's an easy fix for this, as we've never seen these errors before. It almost looks like possibly some Anti-Virus software is killing sessions on your web server during the upgrade - although this is just a guess based on the 'Thread was being aborted" errors.

Do you have any AV Software on your web server? If so, can you try disabling/uninstalling it, then restore a backup of your database. Once you've done this, and restart your browser, it should prompt again to try and upgrade the database.

Regards

Click Studios

Share this post


Link to post
Share on other sites

Thanks for the quick reply  - we successfully backed out as we don't have AV on this server.
 

What are our next steps, to identify the issue?  Is there anything I could provide that would help?

 

FYI - I took a backup of the database (in case it can provide some insight) along with a backup of the inetpub\passwordstate.   I will likely set-up a lab-environment to replicate  production tomorrow (as it is 11:50pm local time).

 

Thanks... Scott

Share this post


Link to post
Share on other sites

Hi Scott,

 

I think setting up a lab environment, with a clone of your production DB would be a good start. If this issue is not environmental, or a once of transient issue during the upgrade, then the easiest way we can debug this is with a copy of your database - although not many customers are willing to do this, which we completely understand.

Let us know how the lab environment upgrade goes.

Regards

Click Studios

Share this post


Link to post
Share on other sites
2 hours ago, support said:

I think setting up a lab environment, with a clone of your production DB would be a good start

 

Of course I sincerely hope that @SGauvin and colleagues are actually doing this: first trying the upgrade on Dev/Test/Acc, before moving to Production ;)

 

 

Quote

It almost looks like possibly some Anti-Virus software is killing sessions on your web server during the upgrade - although this is just a guess based on the 'Thread was being aborted" errors.

 

If I'm reading the linked knowledge base article correctly it's not that there's something actively aborting your connections. It's a matter of the programming language reacting to an error and aborting the thread by itself to handle the error.

 

https://docs.microsoft.com/en-us/dotnet/api/system.web.httpresponse.end?redirectedfrom=MSDN&view=netframework-4.7.2#System_Web_HttpResponse_End

 

As far as I can tell, the threads are being killed because the upgrade application has run into errors with the PasswordState code. Specifically, there appear to be problems with Scott's environment and these specific functions:

 

  • Build_7721_CopyPassword()
  • Build_7721_UpdateNonActiveDirectoryAccounts()
  • Build_7721_Updates()

 

 

Quote

We don't think there's an easy fix for this, as we've never seen these errors before

The "thread aborted" errors have shown up before on these forums:

 

 

Share this post


Link to post
Share on other sites

First to answer the comment from 'Buckit', yes we did test the upgrade in our lab but we did not have a true replica of the production setup (for the lab we had both the Application and DB on the same server. and in production these are on different servers), and we did not have a production copy of the data.   
 

I'm now creating a replica of our production setup (both on virtual-servers).  We've restored the production database into the Database_labvn.

 

We also cloned the production Application server into a Application_labvm and modified the web.config to point to the Database_labvm.   

On the Application_labvm, is there a file where I can modify the URL (https://hostname:9119) so that it points to the Application_labvm and not production (example https://Application_labvm:9119)?

 

Thanks... Scott

 

 

 

 

 

Share this post


Link to post
Share on other sites

Hi Support,

 

I now have an environment that is a replica of production.   I attempted the -upgrade- and it failed in the identical spot:

 

2018-12-05 11:50:42 AM - Build 7721 - Step 2 Completed Successfully.
2018-12-05 11:50:46 AM - Build Process 'Build_7721_CopyPassword()' failed with the following error - Thread+was+being+aborted. ::: +++at+System.Threading.Thread.AbortInternal()%0d%0a+++at+System.Threading.Thread.Abort(Object+stateInfo)%0d%0a+++at+System.Web.HttpResponse.AbortCurrentThread()%0d%0a+++at+Passwordstate.Common.AddDebugInfoData(String+Category%2c+String+DebugInformation%2c+String+EventType)%0d%0a+++at+Passwordstate.Audit.AddAuditing(String+strActivity%2c+String+strDescription%2c+Int32+intPasswordListID%2c+Int32+intPasswordID)%0d%0a+++at+Passwordstate.Upgrade.Build_7721_CopyPassword(String+PasswordID%2c+String+HostID%2c+String+ScriptID%2c+Boolean+ResetStatus%2c+DateTime+LastResetDate%2c+Boolean+AuditRecordFound%2c+String+HostName%2c+String+UserName)
2018-12-05 11:50:46 AM - Build Process 'Build_7721_UpdateNonActiveDirectoryAccounts()' failed with the following error - Thread+was+being+aborted. ::: +++at+System.Threading.Thread.AbortInternal()%0d%0a+++at+System.Threading.Thread.Abort(Object+stateInfo)%0d%0a+++at+System.Web.HttpResponse.AbortCurrentThread()%0d%0a+++at+Passwordstate.Upgrade.Build_7721_CopyPassword(String+PasswordID%2c+String+HostID%2c+String+ScriptID%2c+Boolean+ResetStatus%2c+DateTime+LastResetDate%2c+Boolean+AuditRecordFound%2c+String+HostName%2c+String+UserName)%0d%0a+++at+Passwordstate.Upgrade.Build_7721_UpdateNonActiveDirectoryAccounts()
2018-12-05 11:50:46 AM - Build Process 'Build_7721_Updates' failed with the following error - Thread+was+being+aborted. ::: +++at+System.Threading.Thread.AbortInternal()%0d%0a+++at+System.Threading.Thread.Abort(Object+stateInfo)%0d%0a+++at+System.Web.HttpResponse.AbortCurrentThread()%0d%0a+++at+Passwordstate.Upgrade.Build_7721_UpdateNonActiveDirectoryAccounts()%0d%0a+++at+Passwordstate.Upgrade.Build_7721_Updates()
 

 

What are the next steps to determine the cause? 

 

p.s   I can restore the Application and Database at will, so further debugging can be performed if necessary.

 

Thank... Scott

Share this post


Link to post
Share on other sites

Hi Scott,

 

Sorry this still seems to be an issue. If this is somehow data related, then we would need to be doing some testing with your database in our environment, so we could add debugging into source code to try and determine where the issue lies.

 

Do you think this would be a possible at all - we can sign an NDA if you like to say we will not look at any data at all, but instead just test/debug the upgrade process?

Regards

Click Studios

Share this post


Link to post
Share on other sites

Unfortunately, as you've experienced with other customers, providing a copy of our database is not an option (even with a NDA).

 

Here are a couple of possible options:

1) Is this something that could be facilitated via a WebEx (or something similar), where your team could access our environment, add the debugging code and execute?  In spite of the time difference, I would make myself available

2) As an alternative, provide me with source code with Debugging enabled and I can run it,  then provide you with the results.  

 

Let me know if either of these would work, or suggest another alternative.

 

p.s.  I'm on vacation next week (Dec 10-14), so whatever is decided will have to be executed the week of Dec 17-21

 

Thanks... Scott

Share this post


Link to post
Share on other sites

Hi Scott,

 

Sure, we more than understand. We also cannot share our source code with customers, so we'll need to try and figure this out by sharing the SQL Queries as part of this upgrade function, and see if we can spot any problems with the data.

I will put this together for you over the next couple of days, and email you the SQL queries directly.

Thanks, and enjoy your vacation.

Regards

Click Studios

Share this post


Link to post
Share on other sites

Hi Scott,

 

Good news, and we're really sorry we messed you around with this, but we've been able to reproduce the same error you were getting, and have been able to provide a fix for it. We will release a new build in the next few days, and I will post back here again once that new build is available.

Sorry again Scott.

Regards

Click Studios

Share this post


Link to post
Share on other sites

Hi Scott,

 

Today we have released build 8556 which includes a fix for this.  Thanks for bringing it to our attention and you should now be able to successfully upgrade.  Please let us know if you have any more issues with this!

 

Regards,

Support.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×