What's Going On With Site Performance
- Quick Nav
-
This popup contains your top 10 most visited pages.
Get quick access to your favorite links no matter where you are on Scoutbook. They're just a click away!
Got it...thanks!
What's Going On With Site Performance
In my last blog on 9/14 I acknowledged that Scoutbook has been having some performance issues. Many of you have experienced slow responses when moving thru the site or updating Scout information, and in some cases you have seen the dreaded "Whoops!" message as the site failed to respond to your request. And, the site went down Tuesday evening for a short period of time. This outage was traced to a recently installed module of monitoring software and corrected quickly.
The situation is unacceptable and the Scoutbook team apologizes for the difficulty it has caused our users. The full attention of both the development and BSA Operations teams are focused on the situation. More on that in a moment.
As I mentioned in my last post, Scoutbook has crossed the one million user mark and it currently has almost 20% or packs and troops using the tool. The impact is pretty much a doubling in site volume for users, sessions and page views. While the development team worked in the Spring to anticipate the higher demand we saw coming, we clearly underestimated the growth and its impact on the site.
So, what are we doing about the situation? First, what we are not doing - throwing hardware at it. While tempting, the assessment is that this will only mask opportunities for real, sustainable performance enhancement.
At this time the development team and Operations is 100% focused on performance improvement opportunities. Bugs, site enhancements, new functions, etc. have been set aside for at least the next week to 10 days. This will allow the team, supported by BSA Operations and outside resources to focus on analyzing, identifying opportunities and executing them to improve the user experience. Actions completed or in process to that end are as follows:
- Optimizing a significant number of queries to speed data retrieval.
- Caching more content so that it is readily available to users, rather than requesting it from the database each session.
- Moving from sessions in the database to web server-based sessions, lessening database load.
- Changed the backup schedule to full backup every week, differential backup every day (1am), commit log mockups every 15 minutes - reducing I/0 pressure.
The Operations and Development teams continue to identify additional opportunities for improvement during this performance "spike". We will continue to pursue the opportunities as identified over the next week to 10 day. We are confident that users will see progress. Please keep us posted in response to this blog or on the forums with what you are experiencing. The team monitors these areas daily and while we do not always reply, your comments are an important part of our monitoring and development input.
I would also like to acknowledge that those of you who use AOL for email may not have been received notifications sent by Scoutbook. We are aware of the situation and have reached out to AOL to see how this situation might be resolved.
Gig 'em Aggies!
This is still a problem ref#24-1654311 the scout shows a charge assigned in payments that never was charged it can't be deleted.
Both of these issues were in your court since February and March respectively but nothing was done. These two concern me that is there no way to report an issue that is in fact an issue to be corrected or they just get ignored.
(Also, the first time I tried to post this, I got a whoops.)
Need improvements on the calendar and boards and a real report writer.
Not to mention all the darned forms that still need filled out with information that scoutbook knows. Keep moving forward and don't make it worse first do no harm
This concerns me. Move it to memcache or redis, not to web application layer -- I hope that is what you really meant. That will just cause more login issues unless you use sticky sessions at your load balancer and that is a bad idea for scaling.
It seems like you aren't going to be able to avoid more hardware for long. When will you traffic volume double again? 50% performance increase from software improvements would be awesome but 100%, 200%?
Randy, is it possible it took longer than 30 minutes to compose? I also had an issue with sending a message this morning, but I had gotten distracted in the middle and it sat for a while (possibly 30 minutes). I, however, got an error message popup that the message could not be sent. So, I copied my message, logged out and back in, then pasted the message back in and it sent.
Regarding the discussion of time-outs above, I started a stopwatch last night to see how long my timeout was. I got two different values in my trial: 25 min+/- the first time and 22 min +/- the second. I'm not sure how the timeout is being determined (storing LastActivityTime and comparing against CurrrentTime? Creating a new "LogoutTime" every time an activity occurs?), but it seems like sometihng is behaving inconsistently. A 1-2 minute deviation from 30 minutes seems like a user-timer error. Something like 5-8 minutes seems like something else is going on.
The last 2 weeks have been unacceptable. I convinced our leaders to use Scoutbook, as well as many other units. And the system is completely unusable every single evening when it is needed most.
I am also a software engineer, and I don't understand your aversion to "throwing hardware at the problem". Of course you throw hardware at the problem. Hardware is cheap, and it is the easiest and fastest way to scale. I assume Scoutbook is hosted in a "cloud" or virtual server environment like AWS. Spin up tons of servers. It sounds like both the application and databases need a lot of optimizing. Spin up tons of web servers. Create tons of slave database servers for reading data from.
After you've done that, and the site is actually working for your customers, then it's time to go back to the drawing board and do the optimizations you listed.
A recommendation for something to consider looking into - now that you have instituted stronger security...
Means more authentication attempts as more people are forced to re-authenticate more often. Depending on the approach you use for encrypting/decrypting passwords - this can be a taxing thing to do often. With more and more users doing this more and more often, this is likely to drive up resource consumption.
Jacob Fetzer
Teliah, when you are in the calendar, make sure you click on the gear icon to select which calendar you want to display. You can choose between the unit calendar and den/patrol calendars.
It has been a hard sell to parents and committee members, but some of them are slowly coming around. I would hate for them to turn their back on this tool because of the issues I am coming across. If they can't even log in to view their son's profile, they will probably just give up altogether.
I hope these issues are resolved soon.
This is a fantastic tool and I am excited about applying the information I have recorded on a weekly basis.
Thank you for working on this.
I agree with Mark Gardner, above: we need DAILY updates from YOU on ANY and ALL PROGRESS made or SETBACKS experienced toward resolving this debilitating performance problem!
I have been a paying subscriber for 2 years (i.e., 6+ months BEFORE BSA's acquisition). I have found that communication and transparency has been SEVERELY lacking since BSA acquired Scoutbook. There was a glimmer of hope last year when you came onboard, but that hope quickly diminished when even your self-described "weekly blog" quickly deteriorated into "occasional."
As I'm sure you remember, this time last year, we users were asked for patience when Scoutbook experienced growing pains while being moved to AWS's cloud-based servers. We were led to believe this move would eliminate, or greatly mitigate, the anticipated performance issues due to Scoutbook's growth. This clearly was not the case! You also made reference to how "the development team worked in the Spring to anticipate the higher demand [you] saw coming, [but you] clearly underestimated the growth and its impact on the site."
Please answer me the following:
(1) Why has Scoutbook apparently NOT gained, arguably, the most important benefit with its migration to the cloud -- scalability?
(2) How was Scoutbook's "growth and its impact on the site" so BADLY underestimated last spring? The BSA has owned Scoutbook for over a year and has been clearly drumming up business for it during that time. SB is currently sitting at 20% market penetration, which you've explained greatly exceeded your forecast. Why?!
One would think that "subscription" updates with be a top priority as well?
Please advise if the PayPay/Scoutbook Subscription issue, which was around at the end of 2015 or beginning of 2016, has actually been resolved?
Paypal has to send the information back to Scoutbook saying that the payment has been processed. It is always more difficult when communicating between various systems.
PayPal sends a token back to the originating web site, which when properly integrated like it used to be, should be used to trigger the completion of payment automatically. Looks like this needs to be addressed and PayPay has a great staff to help our Scoutbook developers to correct this basic functionally.
PLEASE LET US KNOW WHAT IS GOING ON - WE DESERVE TO KNOW!!!
Two weeks without a single comment from the "official voice" of Scoutbook is unprofessional, disrespectful to all of your PAYING customers, and just a little cowardly!
This is ridiculous! I am embarrassed that I ever recommended Scoutbook to my pack!
Good luck at fixing it, Last week I was seeing a lot of lag that did not exist over the summer. so far today I cannot pull up my scouts advancements however the lag seems to be gone.
I assume you are already indexing and reorganizing several times a day. and I respect not throwing hardware at it, since that only hides the issues for a very short time on these type of systems.
Please try to appoint someone to keep us informed once a day with a quick blurb on milestones accomplished (that should help keep some angry people pacified).
I can sympathies with the complexity of load testing solutions, and sometimes things do not go as expected when implemented.
once again good luck,
Lots of good points have been made in the comments by other software guys here in the comments.
Your proposed software updates including changing session and improving caching will make a difference, but will only improve performance a small %. I'm assuming you have normalized all database queries by now. Are you running database slave servers that you can run read only queries against?
The real problem we are experiencing is likely CONCURRENT USERS (how many people are on the site in any given moment). Are you load balancing connections across servers?
I appreciate that you are making it a priority to improve the performance. Good luck! I hope that Scoutbook will "Be Prepared" for more growth!
Now, please continue updates frequently (i.e., daily) until these performance issues have been resolved. I just can't fathom why communicating with your user base isn't a higher priority for you?! Must we always beg and plead?
First, I'm not sure if this is a Michigan Crossroads Council/Great Lakes Field Service Council initiative, or BSA wide initiative, but our recharter time is basically here. This is what was stated in our recharter email:
"On October 1, the on-line access will open. All unit charters are to be turned in no later than November 11th. Units that meet the deadline with no defects will be awarded Scoutbook subscriptions for their registered youth for 2017."
Based on the above statement, the 20% increase that Scoutbook is experiencing at this very moment (or is it 20% penetration) could/will increase. My guess is an additional 10%, if this is just Lower Michigan alone, or up to 100% if this is a BSA initiative.
Webelos are learning to not only to "Be Prepared" but as Baden Powell said: "Be Prepared for anything".
Good Luck Scoutbook Team, you have made our unit's life a bit easier, and I'm hopeful that you are preparing for anything too!
Yours in Scouting,
James
I know and appreciate all of the Dev. teams hard work, I am seeing "some" faster page load times. Which is a great thing, considering where we were a few days ago. I imagine that this has been a stressful past week for everyone there.
If I could ask, with all efforts going towards this current issue (as it should) is there a estimate on how far that pushes them back on some of the things that were in the roadmap for release, such as CS Shooting Sports Awards (August 2016), Enable Venturing Subscriptions (4Q 2016), Payment Log Enhancements (4Q 2016), & Advancement Record Sync with my.Scouting/ScoutNet (August 2016)?
Thank you all for your continued diligence, it is so very much appreciated!!
I respect cleaning up the queries to be more efficient, but this is a simple plumbing issue: too many queries in the pipe, you're going to get backed up, and then the: "whoops".
Amazon AWS allows you to spool up a larger server hosting your database for planned times or high demand times. At some point, you will need to throw more hardware at this.
A Scout is helpful, and when you have this many software engineers, and other IT pros who care... Well we just want to help.
Thanks for continuing to keep us posted.
As an aside - the login timeouts are detrimental to authoring emails and other situations. The timing seems to be somewhat erratic. All for security, but convenience is starting to pay the price the way things are configured right now.
Since SB is already on AWS, couldn't fixing the problem be as easy as increasing a presumed cap on virtual servers when demand is high?
It is Cub signup time and the system is failing left and right on multiple issues. Lions prevent sync - I'm now entering 40 new Cubs plus 60 new parents manually. Keep getting kicked off the system. I'm now 10 days behind in the services I promised my families. Just spent an hour writing an email to families only to get kicked out when adding upcoming events. Had to come here to 'vent' as it'll be another 45 min to redo that work. God help me when families discover the flaws.
FWIW - SB needs to put BSA in its place when it comes to the roll out of new programs and awards. e.g. Lions, Shooting Awards, et al. If SB is to be the de facto standard for recording, National must factor that into their timeline.
Keep working at it, there is light at the end of the tunnel. Sometimes it is smarter to find that upper ledge in the tunnel to hang and reassess the situation while the flash flood passes.
Well you have a window and I have to say congrats! Load times are running much smoother after yesterday's two updates. Is it perfect, no, but it is working much better. Please know that your diligence in this matter is greatly appreciated.
Please continue with the swiftness applied toward the performance problems. I still vote to allow an API to allow other developers to help you, but I do understand the security concerns. It seems like there are some very intelligent users who might be able to code reports and other things that can take those enhancements off your plate so you can focus on other things. I think part of being a good stuward with what you have also means using resources when you can. Why not use our wonderful scouting volunteers, and perhaps even scouts themselves.
Keep up the good work!
Do you have a planned deadline for Akela integration completion?
Do you have a planned deadline for the Merit Badge Counselor enhancements?
Do you have a planned deadline for the JTE tool to work?
Do you have a planned deadline for the training reports to be fixed? With training, would it just be easier to take out the training modules code and once linked to the One Sign On for My.Scouting.org just have Scoutbook display the data from the training manager in my.scouting.org showing there official % and the report from my.scouting? BSA spent a great deal of funds to develop the training module in my.scouting, can't we use what we already have?
Again, thanks for all you are doing and please tell the team they did a great job over the last week and we appreciate their sacrifice and diligence.
One thing that I am trying to figure...I can't seem to find links for the Scoutnet link...I seem to recall that almost every page had a link that would sync the Pack information with Scoutnet...has that been moved or modified?
One DESPERATELY needed feature is quick entry for payments due. How do you expect me to add dues for 40 Scouts one by one?
If in fact you did fight the good fight and optimize the code I applaud you. As a developer and IT Manger I know how hard it is do things the hard but in the end more valuable way and more importantly how near impossible it is to find/fix/test performance issues when the overall load is lower. Throwing hardware at the problem would be the easy short term fix but then everyone would be complaining again next year when the issues resurface.
Thanks
Are you also having trouble with the import for a troop, or are both units packs with Lions?
I understand not throwing hardware at a problem, but I think you might need to do something. Wish you all the luck, but have to say it is probably far too frustrating to get anyone on board with it right now. Will review in a year or so again.
Delete Comment?
Are you sure you want to delete your comment?
This action cannot be undone.