Connect with us

Uncategorized

How Segment redesigned its core systems to solve an existential scaling crisis

Published

on

Segment, the startup Twilio bought last fall for $3.2 billion, was just beginning to take off in 2015 when it ran into a scaling problem: It was growing so quickly, the tools it had built to process marketing data on its platform were starting to outgrow the original system design.

Inaction would cause the company to hit a technology wall, managers feared. Every early-stage startup craves growth and Segment was no exception, but it also needed to begin thinking about how to make its data platform more resilient or reach a point where it could no longer handle the data it was moving through the system. It was — in a real sense — an existential crisis for the young business.

The project that came out of their efforts was called Centrifuge, and its purpose was to move data through Segment’s data pipes to wherever customers needed it quickly and efficiently at the lowest operating cost.

Segment’s engineering team began thinking hard about what a more robust and scalable system would look like. As it turned out, their vision would evolve in a number of ways between the end of 2015 and today, and with each iteration, they would take a leap in terms of how efficiently they allocated resources and processed data moving through its systems.

The project that came out of their efforts was called Centrifuge, and its purpose was to move data through Segment’s data pipes to wherever customers needed it quickly and efficiently at the lowest operating cost. This is the story of how that system came together.

Growing pains

The systemic issues became apparent the way they often do — when customers began complaining. When Tido Carriero, Segment’s chief product development officer, came on board at the end of 2015, he was charged with finding a solution. The issue involved the original system design, which like many early iterations from startups was designed to get the product to market with little thought given to future growth and the technical debt payment was coming due.

“We had [designed] our initial integrations architecture in a way that just wasn’t scalable in a number of different ways. We had been experiencing massive growth, and our CEO [Peter Reinhardt] came to me maybe three times within a month and reported various scaling challenges that either customers or partners of ours had alerted him to,” said Carriero.

The good news was that it was attracting customers and partners to the platform at a rapid clip, but it could all have come crashing down if the company didn’t improve the underlying system architecture to support the robust growth. As Carriero reports, that made it a stressful time, but having come from Dropbox, he was actually in a position to understand that it’s possible to completely rearchitect the business’s technology platform and live to tell about it.

“One of the things I learned from my past life [at Dropbox] is when you have a problem that’s just so core to your business, at a certain point you start to realize that you are the only company in the world kind of experiencing this problem at this kind of scale,” he said. For Dropbox that was related to storage, and for Segment it was processing large amounts of data concurrently.

In the build-versus-buy equation, Carriero knew that he had to build his way out of the problem. There was nothing out there that could solve Segment’s unique scaling issues. “Obviously that led us to believe that we really need to think about this a little bit differently, and that was when our Centrifuge V2 architecture was born,” he said.

Building the imperfect beast

The company began measuring system performance, at the time processing 8,442 events per second. When it began building V2 of its architecture, that number had grown to an average of 18,907 events per second.

Lyron Foster is a Hawaii based African American Musician, Author, Actor, Blogger, Filmmaker, Philanthropist and Multinational Serial Tech Entrepreneur.

Continue Reading
Comments

Uncategorized

Wattpad, the storytelling platform, is selling to South Korea’s Naver for $600 million

Published

on

Wattpad, the 14-year-old, Toronto-based, venture-backed storytelling platform with reach into a number of verticals, is being acquired by Naver, the South Korean conglomerate, in a $600 million cash-and-stock deal.

Naver plans to incorporate at least part of the business into another of its holdings, the 16-year-old publishing portal Webtoon, which Naver launched in 2004, brought to the U.S., and that features thousands of comic strips created by its users. It also has a huge audience. According to Naver, Webtoons was averaging more than 67 million monthly users as of last August.

On its face, the deal appears to make sense. According to Korea’s Pulse News, some of  Korea’s webtoons are finding a broader audience and crossing over into film. (Below is a trailer for one popular series called “The Secret of Angel.”)

Similarly, Wattpad, which originally launched as an e-reading app, has evolved into a hugely popular platform where users publish their original work and more than 90 million people visit monthly to read them.  (According to a story published last week in the Verge, Wattpad has published more than a billion stories over the years,  and it claims its users spend a collective 22 billion minutes per month reading these.)

Like Webtoon, Wattpad has been more focused on streaming media, given the many platforms now needing fresh content, from Netflix to Apple to farther flung outfits, like GoJek’s GoPlay, launched by the Indonesian ride-hailing giant in 2019. (In addition to Wattpad Studios, Wattpad also launched a book publishing division in 2019.)

CEO Jun Koo Kim of Webtoon said in a release about the new tie-up that it represents a “big step towards us becoming a leading global multimedia entertainment company.”

Meanwhile, CEO Seong-Sook Han of Naver — whose properties include the popular Tokyo-based messaging app Line — said in press release that Wattpad co-founders Allen Lau and Ivan Yuen will continue to lead the company they have built post-acquisition.

As for whether the acquisition is a win for Wattpad’s investors, it appears to be a moderate one. (It’s hard to discern much without knowing the terms under which each outfit invested.)

Wattpaid had raised $117.8 million from investors in Asia, the United States, and Canada over the years and closed its most recent round with $51 million from Tencent Holdings, BDC, Globe Telecom’s Kickstart Ventures, Peterson Group, Canso, and Raine Ventures.

That last deal, announced in 2018, assigned the company a post-money valuation of $398 million according to Pitchbook.

Continue Reading

Uncategorized

Reflections on the first all-virtual CES

Published

on

I’ve spent more time than I care to mention over the last several years wondering aloud about the value of in-person trade shows. There’s something seemingly antiquated in the idea of jamming a bunch of people in a room, walking from booth to booth. Sure, they’ve fulfilled an important need in the past, but aren’t they just a relic in this hyperconnected world?

I’ve always assumed that if trade shows were to go extinct, it would be a gradual process — a slow fade into cultural irrelevance, like bookstores and record stores (both things I miss dearly). Technology has, for many intents and purposes, dramatically reduced their relative value to our society.

While it’s undoubtedly true that Spotify and the Kindle Store are lacking in much of the appeal and all of the charm of their real-world counterparts, we’re happy to sacrifice all that and more at the alter of convenience.

A rampaging pandemic has effectively given us a year without in-person trade shows. That means, among other things, we’ve had a much more immediate control variable in this question about trade shows. Last year’s CES managed to get in just under the wire. The next major consumer electronics show — Mobile World Congress — was eventually canceled after much hand-wringing.

The CTA (the governing body behind CES) appeared to have been planning a scaled-back in-person version of the show this year, following a similar move by the team behind the Berlin-based IFA over the summer. By July, however, it was clear that such a plan was untenable. To put it bluntly, the United States didn’t have its shit together when it comes to keeping this virus in check (I’d be remiss if I didn’t acknowledge that we just hit 400,000 deaths on the day I’m writing this).

CES 2021 was far from the first tech show to go all virtual over this past year. The size and scope of the event, on the other hand, are relatively unique here. Per the CTA, the 2020 show drew north of 170,000 attendees. The majority of the tech events I’ve attended virtually in the past year have been put on by a single company. CES is obviously a different beast entirely.

The CTA’s (nee CEA) role in the industry certainly afforded it a fair bit of goodwill up front. The show, after all, dates back to the late-60s. It has ebbed and flowed over the years (taking hits from external forces like the 2008 financial crisis), but it has remained a constant. Those of us who’ve been doing this for a while tend to face the show with equal parts anticipation and dread. But the companies always come out.

Per the CTA’s numbers, nearly 2,000 companies launched products at the 2021 event. The figure pales in comparison to the 4,419 companies exhibiting last year, but that’s to be expected. In addition to the uncertain nature of the event, it’s been a remarkably crappy year for plenty of companies. I certainly had my questions and doubts going in — chief among them was the value of an event like this for a startup? Without an in-person element, wasn’t this just yet another chance to get lost in the noise?

I heard similar feedback from startups on the side, though ultimately nearly 700 chose to exhibit at the show. I know because I ended up going through all of them for the purposes of our coverage. It brought back a kind of visceral memory of the year I challenged myself to walk every square inch of the show, and ended up being challenging for entirely different reasons.

Ultimately, this was the element I missed the most. For me, CES’s biggest appeal has been the element of discovery. Eureka Park, the jam-packed startup portion of the show at the Sands Expo, is easily the best part. The vast majority of exhibitors are not for us, but I still get a charge stumbling on something new and innovative I’ve not seen before. The blogger instinct that lives dormant inside kicks in and I can’t wait to get back in front of my laptop to tell the world.

There was no Eureka Park this year — not even a virtual version. There’s just no good way to approximate a show floor online — at least none that I’m aware of. A couple of existing contacts offered to send me stuff in the mail to look out. Sensel, for instance, has a new version of its trackpad (which it announced today will be integrated into Lenovo’s latest ThinkPad). But for obvious reasons, it’s just not possible to get all 700 startups to send review units to my one-bedroom in Queens.

More than anything, the virtual event highlighted the technology limitations of an event at this scale. Press conferences are simple enough (though I found frustration in the various different platforms the CTA employed). More often than not, these felt like lengthy commercials for the exhibiting company. The in-person versions are, as well, of course, but we tend to be blinded by the spectacle. For my own purposes, there just wasn’t a lot that that couldn’t have been accomplished more efficiently with a press release.

The nature of news releases was far more nebulous this year. More companies seemingly took liberties by dumping their news well ahead of the show. Other companies offered their own sort of counter programming. One of the biggest advantages to these events when it comes to my own peace of mind is how they regulate the news flow. I know going into the year that there’s going to be one hair-pullingly difficult week at the beginning of the year where a ton of news is announced.

With CES less of a center of gravity this year, I anticipated seeing a less segmented news flow. I’ve commented to colleagues over the last couple of years that there’s “no more slow season” when it comes to hardware news, and this will likely only increase that sentiment. Obviously there’s upside in having things more evenly spread out, but I’ve got the feeling we’re moving toward something more akin to a series of small CES-like events throughout the year, and the thought makes my blood turn cold.

It’s been clear in recent years that companies would rather break out from the noise of CES in favor of their own events, following in Apple’s footsteps. Virtual events are a perfect opportunity to adopt that approach. Apple, meanwhile, moved from one event to a series of one smaller event every month toward the end of the year. When you’re not asking people to fly across the country or world to attend an event, the bar for what qualifies as news lowers considerably. Perhaps instead of having thousands of companies vying for our attention at one event, we’re moving toward a model in which there are instead thousands of events. The mind boggles.

I have some hyper-specific grievances about the CTA’s format, but I’ll save them for the post-event survey that I may or may not get around to filling out. I still found value in the virtual event. It was an excuse to talk to a bunch of startups I wasn’t familiar with. Ultimately, however, I think the event served as a testament to the fact that as much as we bemoan all of the headaches and head colds that come with an event like CES, there’s still a lot of value to be had in the in-person event.

There’s little doubt that the CTA and the rest of these sorts of organizations are champing at the bit to return to in-person events, even as a bumpy vaccine rollout leaves a big question mark around the expected timeline. There’s a very good chance that we’ll view 2020/2021 as the beginning of the end for the in-person trade show. But given the sorts of limitations we’ve seen in the past year, I’m not ready to declare them fully dead any time soon.

Continue Reading

Uncategorized

Brave web browser adds native support for peer-to-peer IPFS protocol

Published

on

The decentralized tech community is aiming to find support for technologies that go beyond cryptocurrency support.

In a blog post, today the team at Brave announced that they have worked with Protocol Labs to integrate native support for the InterPlanetary File System (IPFS) inside their browser. The peer-to-peer file sharing standard launched in 2015 and has been gathering support among open-source advocates who laud the protocol’s ability to stop companies and government bodies from taking down content across the web, as well as the more functional performance improvements, offline file viewing capabilities and underlying reliability.

IPFS shares plenty of similarities with BitTorrent and allows files to be hosted by a multitude of users distributed across networks. With the update, Brave users will be able to access content from web addresses starting with ipfs:// and will be able to host an IPFS node themselves. The company says that adding support for IPFS will help improve “the overall resilience of the Internet.”

Brave is a likely home for the IPFS protocol given the company’s affinity for all things decentralized. The startup founded by Mozilla co-founder Brendan Eich says it now has 24 million monthly active users. Some of Brave’s most unique features have involved blockchain or peer-to-peer tech. In 2018, Brave announced a beta of Tor Tabs bringing the decentralized Onion protocol into the mix.

Last year, Opera announced that it was bringing limited support for IPFS to its Android application.

Decentralization tech is finding more mainstream interest as tech companies have slowly warmed up to the opportunities in cryptocurrency. Last week, TechCrunch looked into how Twitter was looking to help build out a decentralized network for social media platforms.

It’s unclear whether this is a technology that more mainstream browsers will opt to support natively, given the clear potential for abuse that exists in allowing users to work around file takedowns and the fact that is a pretty niche technology for the time being.

Continue Reading

Trending