Before we start

The Startup Security Story Series

Before this series, I don’t think I’ve ever seen this style of content written down - and I deeply could have used this myself when I first had security responsibilities suddenly thrown onto my plate. I’d have known when to push back more against making bug bounties thinly veiled as ransom payments. I’d have known when to lean in more to 1-on-1 security training so that our sales team would stop sending virtual credit cards to our CEO via text.

This series isn’t meant to be a playbook of what to do. It’s meant to tell the stories of the unsung heroes on the frontlines of startups keeping them safe. These roles are challenging, stressful and often thankless. Our goal is to share these stories, and in the process, help educate others on what you may or may not encounter. If we can help share a lesson or two along the way, amazing. If not, at least it’s cathartic for the narrators.

On confidentiality

It’s important to mention, before we start, that the names of all individuals and companies in the following interview have been replaced with pseudonyms in order to respect the privacy of those courageous enough to tell these stories. The challenges that go on inside of startups, especially security ones, are stories that aren’t often shared publicly because, frankly, they often sound outlandish when written down. Beyond that, venture backed startups are guarded tightly by their investors and their well compensated legal teams.

As such, we will never publish identifying information, and will alter stories just enough to protect our narrators, while maintaining the integrity of the story that we’re trying to tell.

With that known, let’s get into our first story - about extortion, in anything but name.

Interview

Relevant info

Narrators “Morgan” (CTO) and “Alex” (Staff Security Engineer)
Interviewed by Trey for Startup Security Stories

Trey: Speaking of extortion, I believe you two have a very interesting story to tell. So let’s start at the beginning. What was the state of things when you both came onboard? Paint me a picture of the company and the security landscape.

Morgan: Honestly, it was a whirlwind. This story occurred over around a 12 month period. I came in as CTO at a point when the company was scaling up fast, and a lot of the foundational stuff—security, IT, even basic onboarding—was just barely holding together. The board had started to ask harder questions about security posture, and so as I came in I also brought in a VP of security and infrastructure. We were looking to replace our MSP and our new VP found an MSP that looked solid on paper. At the same time, Alex was coming in to replace the previous security engineer.

Morgan: We agreed to bring in this MSP, let’s call them Shielded Sanctuary, for a six month trial run, a premise that we later found out wasn’t totally accurate. Our security VP had apparently, off the record, given a verbal agreement that was along the lines of “hey I just need you guys to come in and give us a huge discount, then I’ll get you guys in and you get a three-year deal at the end of the six-month trial”. The other thing we hadn’t understood was that they’d be bringing their own tech stack.

Trey: The whole thing? Not reusing anything that you had?

Morgan: The whole thing. Alex and I spend a long time trying to figure out what is all that tech stack and why do we actually need it? Um, but that was part of the details as we were slowly uncovering things as they came in. The other piece of context I’d like to share is that our new security VP ended up leaving the company a few months later, I’ll spare you the gory details. So, at which point Shielded Sanctuary, the new MSP, was already “onboarded”, but just like with the super, super basics. And then Alex was working with them to get a basic sort of ticketing process setup up. I don’t know, Alex, do you want to chime in here? I remember those were generally fun times, too.

Alex: Yeah, you know something that you said, Trey, right away that was interesting is that you said a lot of times at startups there’s one person who just gets told to “do security” and that was very much what happened here. It was one of the reasons why, during my interviews, it was one of the things that the company had wanted to bring in and I have a unique resume in terms of doing this at large tech companies with super rigid processes and things like that, and then I’ve also worked for some quickly scaling startups and so I’ve seen both worlds. I’m a pretty risk averse person by nature, and so what I wanted to bring to TarlaTech was a balance between both, because it wasn’t really a startup anymore but it still operated like one. It was a pretty small company, and so a lot of what Morgan and I faced were cultural challenges trying to implement security where previously there had been none vs infrastructural changes. So we were navigating a lot of that. That’s relevant to the story, because the biggest thing that I kep saying about Shielded Sanctuary was that the setup was off. From day one, I came in and was introduced to them by the security VP and I was like “This is weird, I don’t know if they’re hiring me at the same time in parallel with this MSP. But everything that they were offering to do and handle with their tech stack was what I had just been hired to do. So to me, it seemed like there was already a clash from the start, of responsibilities, because it seemed like we were doing the exact same thing.

”A Clash From the Start”

Trey: So, right away, you’re seeing friction. Was this just a matter of personalities, responsibilities, or were there deeper issues?

Alex: It was deeper. When it became just Morgan and I, you know in the partnership, with Shielded Sanctuary we were both super aligned that there was a mismatch of their expectations and what we needed from them. The biggest thing is that, you know, like we said they really wanted their tech stack implemented, not ours. And I just kept warning that, typically, I’d managed MSPs before, right, and typically it’s the other way around. We use our existing infrastructure and bring them into it and give them limited admin access to be IT support. What they wanted was for us to be under their tenants of literally everything.

Trey: Right.

Alex: I know sometimes MSPs do that when they’re operating more like an MSSP where they’re giving us software and licenses, but they’re still not the… they were really adamant about being the full super admins of all of their systems and giving us limited access to it. And I was super hesitant about that.

Morgan: Yeah, I remember Alex around that time frame, as we started to get deeper into just onboarding them, the integration, Alex was calling this out and saying “hey, this is just dangerous. This is risky, and I don’t feel good about it.” We were going back and forth like, okay, what do we do? I don’t feel great about it either. At the same time, there are business agreements, there are contracts, you know, various protections.

Trey: You have that to fall back on.

Morgan: Exactly, and so I was like “let’s keep going”. You know what I mean? That was the idea.

Trey: Right.

Alex: Yeah, and like I said, Morgan is really good about balancing out my risk aversion, too, where, as a security person, it’s kind of your job to think of the worst case scenario all the time, right?

Trey: Yeah, I feel you on that.

Alex: And so it’s easy to get sucked into that, which is one of the themes here, and it’s funny, because, as you know, well we’ll get there, this ends up being a worst case scenario, but in most circumstances it wouldn’t be. And so the biggest learning takeaway for me, about this whole experience, is about balancing that. Balancing where I’m like, yes, I didn’t feel good about it, and I didn’t really love the way that they wanted to set this up. But really the biggest thing here is that if your MSP is the sole architect and administrator of your infrastructure, whether that be security or software, it leads to a very different kind of exposure, but really in that it’s a kind of anti-exposure, where you don’t have enough access and enough control over your own systems and your own infrastructure. So when you’re aligned with an MSP like that, if they are providing you the software and the infrastructure, you need to be working alongside them to make sure that the access is parallel and that you always, no matter what, have access to your own systems.

Trey: Okay, so now you’ve really got me listening about this “worst case scenario”.

Alex: Hah, yeah - so there should never be this scenario where they can remove or block you from your own systems. And that was really what I was looking at was this world where they’re not granting us full admin into these things.

Trey: Did you try to push back? What was the response from Shielded Sanctuary?

”The Slow-Motion Car Crash”

Alex: Every time. I’d say, “We need full access to our own systems.” And every time, it was, “That’s not how we do things. We have other clients in this tenant and blah blah blah.” And I was like, “I don’t care about your other clients, I’m talking about ours”. And it was clear from the very beginning of this, and Morgan can attest to this, they seemed to be used to working only with extremely non-technical clients and were used to them being extremely hands-off and have Shielded Sanctuary do it all for you. We really needed something entirely different, right? We really needed for me to be the IT support, as we already had so much of this infrastructure in house, and it was also clear that they weren’t used to working with strictly cloud clients because they really wanted their tech stack to be things that we didn’t need. Like we don’t need full backup solutions for our assets, because everything lives in our AWS account and things like that. They just really had no idea how to do any of those things that they wanted to with our existing infrastructure and were really rigid to the point that they were unwilling to do any customized solution in any way. That’s where the cracks really started to show, I would say.

Trey: Quick question, was the tech stack more their own custom stuff that that had developed or was it primarily off the shelf?

Morgan: No, mostly off the shelf stuff. You know, whether it was backup, device management or email spam, all those things were off the shelf. It seemed more like they were procuring it themselves at a bulk rate and wanted their clients to sit in their own tenants within these systems.

Alex: Yeah.

Morgan: Yeah, I think as the six months went on, there was certainly friction and it was pretty clear that this wasn’t working very well nor was it a great fit. As we got closer to the end of the term, we started having conversations with their CEO about looking ahead to the future and we were trying to figure out what we were going to do with renewal. Again, I’m going to spare you a lot of the gory details, but essentially, Shielded Sanctuary’s CEO decided that they didn’t want to work with us. And so we were like, “Okay, that’s fine. That works for us.”

Alex: We’d already decided that we weren’t going to move forward with them. We’d decided that the six months had been purely a trial period.

Morgan: Yeah.

Alex: A trial period with them, and so we’d already decided that we weren’t going to keep going after that trial. And we were done with them, and then it was like they decided first.

Trey: So more of a “You can’t fire me, I quit” situation.

Alex: Yeah.

Morgan: Hah, yea. So I was, you know I didn’t explicitly say that we’re terminating or not renewing. But I had been voicing concerns, and those concerns led them to say “You know what? I don’t want to work with you guys, so we’re just done.”

Trey: Oh wow, okay. That’s kind of a quick decision.

Morgan: Yeah.

Alex: Mhmm.

Morgan: You know,it’s fine, I’m not going to get into personal details you know.

Trey: Totally.

Morgan: But so we get an email saying “We’re not going to continue past your six months. How about you go ahead and provide payment and then we will start on your offboarding project.” So Alex and I were quickly like, what offboarding project? So we huddled up quickly, and were confused because we didn’t really onboard that much of anything really, so what’s the point of offboarding since we didn’t really use anything outside of the device management platform that they used. I forget which one that was, ninja?

Trey: Ninja one?

Morgan: Yeah, that one.

Alex: Yep.

Morgan: We liked Ninja, we’d already been using it. So essentially all we needed was for them to run a script to reverse an earlier migration that they’d done to move us back to our own tenancy.

Trey: I mean that sounds pretty simple.

”Locked Out, Locked Down”

Morgan: Right? That was really it. Their person dug in and he basically locked us out of our Ninja instance completely.

Trey: Oh.

Morgan: And then refused to provide access or do anything until we provided payment. Now this payment was kind of fabricated, you know, a payment to do an offboarding project.

Trey: Like just for that project? Okay.

Morgan: Yeah, the offboarding project, which I’d never heard of before. So anyways, like for times sake, we’re basically getting to the point, like the lesson learned is that some people are willing to go outside of contract terms and really push the boundary. I think that they were really dug in and were willing to go at least, or wait until we took legal action. We had other stuff going on, like this was not something that was going to take our legal counsel. I was not going to pull our CFO and legal counsel into it to get involved.

Trey: Okay, so what did you do?

Morgan: We did wind up getting access back, but we had to get our fractional CISO involved because this was now becoming a security incident.

Trey: I mean, yeah, at that point you don’t have control over your devices. It’s like a ransomware incident by your ransomware protection vendor.

Alex: Exactly.

Morgan: Yes, yes. I mean, I can laugh about it now.

Alex: Haha, yea.

Morgan: …it was, yea.

”Security Is a Team Sport”

Alex: It’s a little funny. But it is another one of those situations where Morgan is so good at staying calm in this instance, and I’m not a calm person. I was losing my mind about this because we kept meeting and our fractional CISO was aligned and I was like “this is extortion”. And Morgan kept saying “don’t say that word out loud” and I was like “but that’s what’s happening!”

Trey: I mean, that is what it sounds like.

Alex: Fortunately, they didn’t do anything malicious, and I don’t even think that they meant to actually lock us out. It wasn’t like they put a lock on all of the computers.

Trey: Right.

Alex: But they removed all of our admin access and so the theoretical situation was I had no admin control over any of our assets, and that really was the problem. Again, I don’t think that they were intentionally being malicious, but they kept just saying “we can’t add you back into our Ninja because you no longer are working with us.” And I’m like, I understand that, but you need to put us back into ours. Because, again, we’d already been using Ninja One, but their support engineer wrote a script to move tenants.

Trey: I’d be so frustrated.

Alex: I was saying, can’t you literally just reverse it and have it go the other way? And they kept saying, well this is work for the engineers, it’s an onboarding project. So I said how about I write the script and send it to you, you just have to deploy it - like someone has to click the button on your side because you removed our access for it. And that’s why I was going insane, because I kept thinking “if this person gets more and more mad, they can lock all of our computers” and I would just have had no idea what could have happened. It was just the lack of insight and the lack of control that was the incident here.

Trey: Yeah.

Alex: That we lost admin access to our machines was the issue, and fortunately nothing happened. But it’s one of those things where, if you’re a security engineer, that’s a worst nightmare thing. And so that was my biggest concern in the beginning, that we needed a way to always ensure that we were the admins of those systems. So it could have been way worse, and in a scenario where they actively act as a malicious actor, it could have been way worse. So you know, I think the takeaway there in a startup environment is like when you’re finding your partnerships from the beginning, that you have a clear definition of the responsibilities and knowing what you need out of your partners. I would say that’s Morgan’s point about the contract terms. The one that the original security VP had signed was pretty loose. A lot of those things weren’t strictly defined, which is why they were able to kind of go outside of it and say “well, this is what we have said about offboarding.”

Trey: Absolutely. I mean, that’s such a helpless feeling. What if a device gets stolen or you need to wipe it? What if a critical issue like a vulnerability comes out? You can’t patch it or roll out anything, you’re just sitting there in that time period.

Morgan: Yeah.

Trey: I can also kinda see how from their side, they’re trying to build a good relationship and then things just really start going south on all sides. And I think that’s another hard thing for security people early in their security career, too - you need to know your company’s legal person, your legal team. They are your best friends in those situations to give you that support to help fall back on. But that’s a terrifying situation and I’m sorry that that happened to y’all.

Morgan: Yeah, but you know, we can laugh about it now.

Trey: Oh yeah, it’s a great story afterwards, right?

”We Survived, But We Learned”

Alex: You know, it was also a good lesson to me in being diplomatic. I think working with Morgan, that’s one of the big things I’ve gained - is just how even keeled they are in that situation. And they were also correct in pulling in the right people, I wanted to escalate and pull in our legal team and it would have gotten worse. Whereas Morgan was still looking for a way out of the situation without pulling that card and saying, “well here’s our cease and desist” or something like that. And so being calm, and the ultimate thing Morgan was focused on was that the amount of money was fairly small for the grand scheme of things and was like “is it worth it to keep fighting on this or is it better to just say, all right, whatever.” So I think it was a good takeaway, as well, that in the security world fighting fire with fire doesn’t necessarily always work. Sometimes you have to find a path to deescalate.

Morgan: Yeah, it’s one of those things, moments where like I had to absolutely just swallow my pride and authorize the payment. Because it was unjustified, but for the business, in the grand scheme of things, it was the right thing to do because there’s always a whole bunch of other stuff going on. So I felt I had to make the right business decision, to swallow my pride in that moment.

Trey: Yeah, I understand that.

Alex: Whereas I wanted to burn them to the ground on Morgan’s behalf. I was so mad about this whole thing.

Morgan: I mean I was also insanely mad, it’s just it wasn’t going to come out externally, you know.

Trey: I mean, you’re not always going to solve it, right? You have to do what’s best for the business, and going to resolve the issue permanently.

Alex: Yeah. Again, I’m super grateful that Morgan was there for all of that. And yea, like they said, our theory was kind of that there was no malicious intent, but the whole precedent of the initial, false contract pretenses started this whole situation.

Trey: And at least it ended, and y’all were able to be done with it.

Alex: Yeah.

Morgan: Hah, yeah.

Reflection

Thankful

Again, I’m incredibly thankful to Morgan and Alex for sharing this story - it’s the type of nightmare that lurks in the back of your mind, but you try to convince yourself it will never actually happen.

Lessons

Personally, several learnings that both Morgan and Alex learned resonated with me from my own battlescars.

1 - Be levelheaded

This sounds so simple, it’s hard to write it without laughing, but it is also so incredibly difficult to do in any elevated situation. Calm isn’t what’s on your mind when an existential risk has reared its head on your watch. I have nothing but respect for Morgan to keep their calm when they suddenly found themselves without operational control of their staff’s devices.

2 - Beware MSPs who do everything, without including you

While on paper a “we do everything” MSP sounds great, it only is if you can see what they’re doing. Obscurity never provides legitimate security, and vendors are meant to be partners, not opaque service providers. MSPs provide large value to businesses of all sizes, but the large market space does mean that there are less scrupulous vendors out there. Be diligent in your research, and make sure you understand how you will all operate collaboratively.

3 - Befriend your corporate legal team

You will spend more time with your company’s legal team than you could ever imagine. Get to know them, understand how they work and help them understand how you work. In particular, as this story highlights, try to get everything that you can in writing - this is a major collaboration between legal and security. Security knows what safety rails need to be in place, and your legal peers know how to get that in writing - both parts are needed to protect everyone.

Outro

I deeply enjoyed spending time with Morgan and Alex. I sincerely hope that this story helps someone, even if it only gives you a laugh this week. We’re trying to democratize security at NovaCove, and sharing stories of what actually happens at startups is one of the many ways we’re looking to help do this.

Have suggestions of what you’d like to see? Let me know!

Share your story

Have your own security story? Reach out to me on LinkedIn or at trey@novacove.ai. We protect your anonymity so you can help others learn from the real frontline.