Defence Against The Dark Arts Credential Stuffing Attacks

Defence Against The Dark Arts Credential Stuffing Attacks

Defence Against The Dark Arts

Credential Stuffing attacks are where an existing list of usernames / email addresses and cracked passwords are replayed against your site by an attacker to perform Account Take Overs (ATOs).

 

The list of usernames / email addresses and their passwords are available on the dark web, and often originate from previous Database breaches like linked-in where the old ciphers used to encrypt passwords were weak and so have been cracked resulting in full login details for list of millions of users being available for replay.

 

Because so many users reuse passwords – replaying this list against your site will result in some valid logins for the attackers – and the they exfiltrate any personally identifiable information (PII) from the users account to then potentially sell.

 

PII are adollar or two depending on completeness – but if they can harvest 1000s or millions of PII entries by accessing that many accounts on your site – it’s lucrative business.

 

Various tools such as sentry.mba exist for performing these types of attack – and if the infiltration is ‘low and slow’ it’s very hard to detect a few failed logins per second amongst all the noise

 

Prevention / Preparation

 

Keep max login retry limits low (though in credential stuffing they have a username/password combo already this is minimal protection).


Make sure you rate limit based on sessionIds add a Captcha and reverse-Captcha by default on login, you should ‘feature toggle’ Captchas so it doesn’t have to be on all the time annoying your users.


If you want to get sophisticated you can make this risk based – so the Captcha only springs up when some aspect of the user is suspicious.(Google’s ‘recaptcha’ is pretty clever and non intrusive these days and does risk profiling too).


You still need a reverse capcha too though as this can be left on all the time as it’s invisible to users – there may be circumstances where product folk want to lift the ordinary user Captcha
ensure you’re returning 401 and 403 HTTP codes appropriately in your login validation code (this will facilitate efficient firewall rules).


if attacked you could set some irules for repeat 401/3 from a single IP be a ban on the Ip for 10 minutes for example, this not so easy if you just 302 to /failedlogin.html as you’ll have lots of other ligitimate 302s


Make sure you consider you APIs too – an API gateway of some sort should offer throttling mechanisms


Consider something like Cloudflare, Ravelin or their competitors – they do lots of profiling and filtering of supicious IPs and tools, or ATO plugins and may solve your problem invisibly upstream


Make sure you’ve built the capabilities for Password Reset and Account Suspension for corralling any compromised accounts.


Implement user device profiling, where each new device a user logs in from is recorded and exceptions flagged to users for validation – numerous libraries exist for this
Ensure you close the loop in verifying users email addresses and if possible phone numbers too.


Implment 2FA (2 factor authentication) or MFA ( Multi Factor Authentication) – the gold standard of login verification where users have some kind of key generator with them 9Authy / Google Authenticator Yubikey).

Though this may not be applicable for high volume sites with non technical users, offering it gives you options for expert users.


Do a red team simulation for credential stuffing to see if it gets spotted by your team.


During registration, consider a spot check on new credentials to haveibeenpwned.com or similar — and potentially warn them about strong unique passwords if their username/email is known (careful messaging required!)


See also OWASP’s guide on this at https://www.owasp.org/index.php/Credential_stuffing

 

Detection

 

You need separate monitors / alerts on:


failed logins


A spike on failed logins is a clue for brute force and credential stuffing
logins for non existent users,
A spike in logins for non existent users is a clue for credential stuffing – these are the mismatched emails / usernames you don’t have on your system being stuffed in
have a long look at both failed login monitoring alert sensitivities, assume the worst on any prolonged tickle.


successful logins


A spike in successful logins only could be a clue someone’s stolen your user credentials from your DB (though why they’d then swarm your login page is a mystery)

Accounts locked due to repeated failures
again indicative clue to brute forcing


Any other key metrics you capture around login or account traversal (changing card details, changing user email addresses) should also be emitted and measured for unusual activity
these are post breach, so not ideal – but still very worthwhile in case you miss something subtle initially

The detection thresholds on these login attempts alerts should ideally be sophisticated and use correlations, anomaly detection and machine learning – it’s the only way to really spot the low and slow subtle attacks

 

When under Attack

 

Turn on Captcha


Look for patterns in the attack, are there clues to specific hack tool – perhaps repeating user agent strings, odd protocols like TLS1 or timings that you may be able to profile.


you can set some irules for repeated 401/3s from a single IP to ban on the IP for 10 minutes for example.


Dump suspicious IPs from threat intelligence reports or with high volumes altogether


Or rules to filter on unusual user agents / matching TLS profiles (though this may collateral damage or shedding some real users)


Block IPs outside your customer geography location if possible, any other way of shedding suspicious traffic?


Check threat intelligence sources looking up the dodgy IPs you see for more info


Slow down logins, irritating for users, but throttles the attack


Are accounts being compromised, if so is there a pattern – can you get ahead if so?


Try to catch the traffic as lose to the perimeter of your network


Check this isn’t just a feint for something worse. Check your other alerts for signs of intrusion to backend systems while they’ve kept you busy chasing this login tidal wave.


Call the Police
Call the FBI / GCHQ / Cyber authority in your geography

They can perhaps perform ‘Upstream Disruption’:
Shutting down sites the tools being used are deployed on
Identifying and Arresting individuals performing the attacks
Throttling traffic at ISPs or even further upstream

 

Management

Have an incident plan ready – it’s a fair bit of planning but pays off in a crisis.

Comms is key:
Internal folk (tell them don’t say too much)
Different customer classes (tailor)
Regulatory bodies in your industry

Your Suppliers
be careful of any precedent for refunds / goodwill
Consider getting hack insurance in advance
Police – report as a crime
Gchq / FBI
involve your legal team and have them run their eye over comms.

Controlling AWS Costs

Controlling AWS Costs

4.7/5

Controlling AWS Costs

As you move your estate onto cloud inevitably the cost genie escapes the bottle as engineers and ops personnel of all flavours spin up test and Dev environments and the general number of machines escalates beyond all your estimates and predictions.

 

1) Get your AWS tagging correct from day 1

 

This is an essential step to allow you to slice and dice your costs and see where the money is going
Tagging needs to baked into your Dev Ops process from the start so it cannot be circumvented and is 100% consistent across your estate.


Tagging meta data on instances allowing you to see what’s going on with billing : see

http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/cost-alloc-tags.html

 

2) Make the Teams accountable

 

Centralised cost control is tricky as problems get seen too late in the day, and not by the DevOps folks in the teams who made those technically nuanced and expensive decisions.


Making the teams accountable for their spend is the best solution to this but necessitates


i) their engagement


ii) Visible reporting ( see ‘Show me your numbers’ below)


Ideally this can be slightly gamified by making costs a metric for teams – but fair comparison is tricky so aim for a ratio based metric – perhaps something like
AWS Cost in Cents / Number of Customer Sign ups
with a target of 50 cents per customer a target for all teams

 

3) Show me your numbers


To allow your team to be accountable, and also for an overall view of costs you need to make those costs visible.


Your AWS console has good visualisation tools that allow you to slice and dice based on tags : but making that global costs data accessible to everyone may not be politic – so other Tools like


Netflix Ice (https://github.com/Netflix/ice )
Splunk AWS costs plugins ( https://splunkbase.splunk.com/app/1577/ )
allow a degree more reporting flexibility .

 

4) Get the basics right


Use the AWS billing Alarms
http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/free-tier-alarms.html

 

They are not very fine grained, and need frequent maintenance as your estate grows so as not to mis-alert, but an email warning that you’ve spent 75% of you budget on the 3rd of the month is a very worthwhile exercise, especially in the early days of your use.
Budgets for teams are also supported


http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/budgets-create.html

 

5 ) What are the main causes of waste and possible solutions?


Main culprits for where money gets wasted are

i) Dev test environments on when unused
— Solution: See Step 9) below on a Script to disable Dev environments.


ii) Defunct resources not being terminated on production, load test , staging etc
— Solution See Step 8) Below for Occasional manual sweeps


iii) You’re not using reserved instances
— Solution: Use them – but be careful – See step 7) below


iv) Low utilisation (tends to be the hardest to Solve – see step 6) below)


.a) Machines the wrong size for the work they do
— Solution: Devs need to re-evaluate
.b) Mis-design
– Instance just runs crons twice a week
+ Solution: Perhaps use AWS Lambda based utillty which will spin up resources only as it runs rather than lying idle waiting to work.
– Only one Application per instance
+ Solution: perhaps you need better scheduling on your PaaS to stack multiple Apps on bigger instances?
But bear in mind an average utilisation above 5% is pretty good across a data-centre

 

6) Tips on Low Utilisation


Very often the biggest culprit for wasted cash looks like ‘low utilzation’, meaning the machines aren’t doing much – perhaps just running intermittent batch jobs, or the machine chosen is over-specced .
Unfortunately getting to the bottom of each of the under-utilizations is hard because
i) Each case needs individual investigation (and reasons why things are as they are may be hard to find or long forgotten)
ii) Fixing it may after the event often seems more expensive than what it’s costing
Tooling can help
The AWS ‘Trusted Advisor’ tool on your AWS console is a great and free way to get clues as to where the money is going.
Third Party Services like Cloudyn, Cloudability etc. can also help here as they automatically trawl the AWS cloudwatch logs with some intelligence to recognise common anti-patterns and then make recommendations. This intelligence is something you need to apply yourself if you work on the raw Trusted Advisor data, and there’s aquite a bit of judgement involved
These services typically work on a %ge of your spend though so get your costs house in reasonable order first before giving them 2% of the total.

 

7) Tips on Reserved instances


Reserving instances means committing to them for longer periods of time, but savings can be significant : 30-50%.
It used to be the case that AWS reservations had to be paid for up-front which was a huge outlay, but that’s been addressed now with ‘no upfront cost reservations’ so, if you’re sure an app will stay on a specific machine class for a year’ reservations are the answer…
i) Rightsize, then Reserve. Check your utilisation is good for the instances / applications you’re going to reserve – otherwise you’ll lock in over-specced instances
ii) Start Slow – reserve a few instances and check over the next month that utilisation of those reservations is good and that your process of identifying good reservation candidates is working.
Identifying candidates needs expertise on the estate and the applications so might best sit or at least get reviewed with some centralised dev-ops or architecture function
iii) The reserving interface is a nightmare and Amazon are wary to take-backs on reservations even those done in error so to avoid incredibly expensive mistakes:
.a) 2 heads are better than one – so pair when using it to avoid expensive mistakes !
.b) Submit the reservations form in small chunks – e.g. only reserve 2 line items and submit not 20 and submit – this makes it easier to double check it.

 

8) Make a destroy script to tear down all test / dev environments overnight and weekends


AWS Lambda is a good fit for this.
If your tagging is correct and un-circumventable then you should have the confidence to run such a script, which basically uses the AWS API’s to list resources then turns off everything tagged with e.g. ‘env’ tag set to ‘dev’ or ‘test’.
Make sure as you design such a script you include a ‘white list’ feature where exceptional kit can be configured not be turned off. And also make sure you honour any ‘Termination Prevention’ tags.
The harder bits can restarting stuff in the morning, which probably also needs automation, especially since getting permissions right to make this easy for teams to do this themselves is tricky.

 

9) Occasional Manual Cleanup Sweeps


Clear up any orphaned AutoScaling Groups, Load balancers etc. Sometimes these don’t clear up fully when the pipeline ‘destroy’ step fails for whatever reason.
Remember to tear down load testing environments promptly.

 

10) Be careful with health-check rules on Auto Scaling Groups (ASGs).


Prefer the EC2 status checks to the ELB based ones . Any flaws in ELB rules can lead to machines spinning up in the ASG, immediately being flagged unhealthy and then torn down – over and over. In the few seconds they are up – you get billed for an hour so this fluttering can get expensive.
Part of your reporting in step3) above could be the number of instances used in a day – that will show fluttery instance starts like this

 

11) Prefer AWS Lambda for cron type tasks rather than a dedicated machine.


it’s much cheaper (and cooler), and a dedicated machine for cron often has very low utilisation.

 

12) AWS can bill in numerous currencies….

 

So if you’re outside the US but paying in Dollars there might be some small savings to be had in paying in local currency and tax treatments. See


https://aws.amazon.com/blogs/aws/new-set-preferred-payment-currency-for-your-aws-account/

 

gro.team want to make the world a more successful place…person by person, company by company. If you need an high impact Interim to deliver your change agenda give them a shout..

How To Create A Killer CV/Resume/Profile

How To Create A Killer CV/Resume/Profile

How To Create A Killer CV/Resume

We create and receive a lot of CV/Resumes/Profiles at gro.team so I guess we’re getting pretty good at judging what a great profile looks like, and what helps the author really stand out from the crowd.

 

Job boards, LinkedIn etc can be a good source of jobs but they don’t filter very much (both ways) so how can you rise above the noise in people’s inboxes/feeds?

 

I’m going to apply a lot of what Amine Bellaoud talks out in his excellent article about neuromarketing here because at the end of the day creating and presenting a CV/Resume/Profile is Marketing.

 

Make The CV/Resume/Profile Look Err…Nice

The CV/Resume/Profile is a marketing document so it needs to be well presented and look err…nice.

 

If you can’t be bothered to present yourself well in such an important document can you really be trusted to represent someone else’s team or company?

 

Below is one of gro.team’s interim cto profiles and one that isn’t.

 

 

 

What do you think? Which one looks more interesting and makes you want to read it?

 

The Use Of Photos

I know not everyone will agree but I think profiles should include photos.

There is a lot of evidence to suggest that a photo of a person’s face is the thing most likely to get our attention in a stream of information.

It could be argued that the use of photos increases the potential for discrimination though so I wouldn’t necessarily see it as a negative signal if someone chose not to include a photo.

 

Use Colour, Contrast and Differently Sized Content

Three factors (colour, orientation and size) are additive in terms of getting our attention. (Nothdurft). More factors will grab more attention so ideally you should use all three.

 

Testimonials aka Recommendations aka Reviews

All the gro.team profiles include testimonials for very good reasons.

There is a lot of evidence to suggest that people trust online reviews as much as personal recommendations. It is very important to present recommendations as early and often as you can.

It is pretty straightforward to ask your connections for recommendations on LinkedIn these days.

 

The Three Boxes Of Greatness (™)

 

At gro.team we review profiles against three main criteria of

 

⍏ “Technical” Skills

Impact Focus

Being A Team Player

 

“Technical” Skills

“Technical” skills are the competencies you use to perform a role. If you’re a Developer they might me coding in a specific language, if you’re a Marketeer they might be paid for search, if you’re a scrum master they might include running stand ups or whatever. We have sections like “Expertise” in the gro.team profiles to highlight technical skills. You need to make sure you present evidence that you have the necessary tools in your tool box for the role but don’t overdo it…see Impact Focus below.

 

Impact Focus

It’s really important that you present as someone who wants to have an impact for a team/business focuses on the “what” not the “how”. Yes it’s great that you know how to run Google PPC campaigns but what results did you achieve with them? What were the business results of what you did? We have “Delivery” sections in the gro.team profiles to highlight these achievements.

 

Being A Team Player

These days ticking the other two boxes but being what Netflix CEO Reed Hastings called a “brilliant jerk” will stop you getting hired by great companies. As he said: “Some companies tolerate them. For us, the cost to effective teamwork is too high.” We have sections like “Relationships” in the gro.team profiles so that we can demonstrate that being a good team player is important to the people in our network.

Make sure you tick all these boxes in your profile.

 

How long should the CV/Profile/Resume Be?

All the gro.team profiles are a maximum of three pages long and are normally only one page.

We never, ever present a profile that is longer than that.

Why? Because we need to show we can summarise the important information in a context and not just present a stream of uncurated facts.

I guess two-three pages is OK but I would argue any longer than that is counterproductive. Show that you have taken the time to write a shorter profile.

 

Which Format Should You Use?

Our profiles at gro.team are normally in Adobe PDF format. Nearly all platforms and devices can read a .pdf file these days. (Google can and does index .pdfs but it definitely prefers other formats like HTML if you are a creating a document specifically for search engines).

 

So to write a killer CV/Resume make yours look good, use testimonials, keep it short and tick the the Three Boxes Of Greatness…

 

If you create a CV/Resume/Profile that is interesting to look at, ticks the “Three Boxes Of Greatness”, is concise and informative you really maximise your chances of being selected for the next stage of the hiring process.

 

Good luck!

 

Rorie Devine

Share on facebook
Share on twitter
Share on linkedin
Share on email
Share on whatsapp
Share on pinterest
Share on tumblr

I am an inveterate Interim CIO ¦ CTO and the author of The CTO ¦ CIO Bible (#1 rated CTO book on Amazon) and How To Be A Hero CIO ¦ CTO (#1 rated CTO course on Udemy). I write about big ideas and share leadership stories that make you go “A Ha!”

My work has been described as a new genre of leadership advice…it’s based on unrivalled experience, genuine credentials, wisdom distilled down into 100’s chunks of actionable insights and even the odd bit of humour…

Some people don’t agree with what I say. Other people say I changed their career. Why not read/watch/listen and decide for yourself?!?

   85% 5 Stars CTO ¦ CIO Bible eBook & Book

The #1 rated CIO ¦ CTO Udemy & THINKIFIC Course

Fi
Fi
Ticks all the key elements...
Read More
I loved this book although the first few pages initially caught me on the back foot as it is so different from what we are used to. Rorie has pulled together all the major elements of outstanding IT management and delivery by referencing the source material with his own explanation of why it is useful. So this is truly a read which lets you tick the boxes you already worked on and see what else might be useful out there. Personally I found it highly energising and a great read, I hope you do too.
Hutch
Hutch
Insightful and an easy to read
Read More
I’m not a CTO or a CIO but I work closely with many that hold those positions, and this book has given me a wonderful insight into their world, the challenges they face and strategies/tactics they can deploy to be great. The book is easy to read and digest and incredibly informative. Thank you Rorie!
Thomas Kiessling
Thomas Kiessling
A great cookbook
Read More
I highly recommend this book for both CTOs/CIOs at the beginning of their career and for experienced technology leaders on a quest to continuously improve and re-calibrate themselves. The method of the book is to distil decades of experiences into simple learnings. It’s a cookbook. It creates a mental grid of domains and principles for a technology leader to succeed. It also provides entry points to deepen your knowledge in specific subjects, always pointing back to the larger context. Well done. Fun to read.
LWS
LWS
A Fantastic Go-to guide
Read More
Very interesting book which provides you with key advice as to how to manage your company and teams as a CTO or CIO. It is solution-focussed. It’s easy to read and can be read in any order. So if you need to know something quick, it’s a great go-to guide. The good thing is that it works on key aspects of management and provides you with fast and straight-to-the-point advice. Highly recommend to professionals with little time on their daily schedules who need a book which they can consult quickly to find a solution or improvement to their current situation.
Mr. Faisal Ahmed
Mr. Faisal Ahmed
Really insightful
Read More
Easy to read in a complex world full of buzz words, really recommend this to anyone in the business field. Cannot wait for a follow up book.
Jonathon Moore
Jonathon Moore
Super!
Read More
Bucketloads of great insight across the full spectrum of product development. A great read for product and engineering leaders alike.
Roseland
Roseland
Very readable
Read More
A very easy to read and digest book that gives lots of tips and ideas for how to make a high impact in a positive way as quickly as possible. Not a technical book, but one of leadership, relationships and how to add value. A definite must for the library
Martin King
Martin King
Thinking from an expert!
Read More
Clear and concise advice for people that already have a lot of the building blocks in place. The book offers plenty of critical thinking from a seasoned expert. Thanks Rorie!
Mr. S. Edwards
Mr. S. Edwards
Wisdom for busy execs
Read More
I loved the simple, concise format of this book. Full of bite sized wisdom and instant takeaways to 'try now'. Rather than a massive tome of boring management theory, busy execs can read this book on a train journey and get instant value. Highly recommended.
manub
manub
Actionable leadership tips
Read More
I've had the pleasure to work with Rorie and I was looking forward to read his book. I wasn't disappointed! The format may not be liked by everybody as more than narrative it's a series of tips, however I find the insights very helpful. On some topics I have slightly different views but Rorie gets his argument through explaining why he's giving those tips, based on his 20 years experience. Definitely a good read.
Alex Farr
Alex Farr
CIO UK Top 100
Read More
An essential read for any #CIO or #CTO in 2020. Authored by the extremely talented Rorie Devine, the book gives the mission, objectives, strategies and tactics needed to succeed. It may even contain a small contribution from yours truly...
Oliver Bayes-Shelton
Oliver Bayes-Shelton
CTO
Read More
Great book by Rorie Devine, I always enjoy reading things by peers! "The CTO ¦ CIO Bible: The Mission Objectives Strategies And Tactics Needed To Be A Super Successful CTO ¦ CIO"
Eoin Woods
Eoin Woods
CTO at Endava
Read More
Interesting new ebook from Rorie Devine on how to be an effective #CIO based on his experience in the role and also working as an interim, seeing inside many organisations. Structured into bite-size pieces for easy digestion!
Michael O'Gorman
Michael O'Gorman
CTO/VP Engineering
Read More
Just read CTO ¦ CIO Bible by Rorie Devine - great valuable insights by very experienced CTO - must read for all current and aspiring CTO’s/CIO’s - great job Rorie Devine
Artur Grabowski
Artur Grabowski
Technical Director
Read More
Some really good tips and advice in "CTO | CIO Bible" by Rorie Devine. Definitely worth reading by anyone involved in a technology leadership role.
Nikos Kryvossidis
Nikos Kryvossidis
CTO at Racing Post
Read More
A very interesting book has just been published by Rorie Devine => http://bit.ly/CTOCIOBible Highly recommended!
Ian Woosey
Ian Woosey
CIO Clarion Housing
Read More
Some really excellent thoughts in here, well worth a read by anyone involved in a technology leadership role. Thanks Rorie.
Rich Phillips
Rich Phillips
Consulting CTO
Read More
Not a long book but, packed with sound advice and tips from the author's own deep and varied experience as a consulting CTO. Reading a book cannot make a person a great CTO or CIO, but a good one like this can certainly help put the aspiring professional on the right path, act as a starting point for further research, and help avoid the common pitfalls.
Louise
Louise
CEO Energy Trading
Read More
Found this book really helpful. Lots of ideas and advice that got me thinking as well as practical examples of successful strategies and advice on how to manage complicated projects. Highly recommended.
Loana Wurzel
Loana Wurzel
COO
Read More
Brilliant! Fantastic insight into what it takes to be a successful CTO or CIO. I love the way it is set up in easy to read sections and it works really well on a phone. It even manages to bring humour into being a CTO which must be difficult. Read it even if you're not a CTO any modern leader would benefit from reading this book!
Ade McCormack
Ade McCormack
Digital Readiness Institute
Read More
Congratulations Rorie. Nice format. Easy to read. Clear direction.
Previous
Next