mallyanitin

Vietnam

Vietnam had always been on my bucket list, but we hadn’t planned to visit it anytime soon. However, as 2024 drew to a close, perhaps inspired by the spirit of “revenge travel” or a spur-of-the-moment decision, we decided to make it happen—just in time. Our hectic work and travel schedules in November and December had left little room for early planning, but the allure of Vietnam proved irresistible, and we finally set our sights on this incredible destination.

This destination was initially tagged as a “plan-it-yourself” trip, but the last-minute nature of our decision made us turn to a holiday agent—MakeMyTrip. They were highly responsive and efficiently planned the key attractions for us, ensuring we didn’t miss the important highlights.

We managed to get our e-VISA just in time, though we were already considering a Plan B. The process typically takes 5-8 days and requires specific documents: a scanned, colored copy of the passport’s front and back pages with at least six months of validity, a recent photo (35 x 45 mm) with a white background (no visible teeth or glasses), and a PDF of the round-trip tickets. While it was a scramble, the heavy lifting and follow-up were expertly handled by the MakeMyTrip VISA team, leaving us with nothing to do but worry and wait.

While there are direct flights available, our last-minute travel plans meant we had to reach Hanoi with a layover in Bangkok.

Day 1: Hanoi – Where the Old Meets the New

We stayed at the Hanoi Daewoo Hotel, situated at 360 Kim Ma Street in the Ba Dinh District. The hotel provided a peaceful retreat in the heart of the city. Although it isn’t located directly in Hanoi’s bustling Old Quarter—approximately 4 to 5 kilometers away in the Hoàn Kiếm District—it offers convenient access to key attractions. A quick Grab ride, typically 15 to 20 minutes, easily takes you to the Old Quarter. After a tiring overnight journey with a layover in Bangkok, we were eager to rest yet excited to start exploring Hanoi.

Lotte Center Skydeck

Since we couldn’t get an early check-in, we started our day by visiting the Lotte Center Skydeck, conveniently located near the Hanoi Daewoo Hotel. The Skydeck offers breathtaking panoramic views of the city and is a must-visit for its stunning scenery. The highlight is the Skywalk, a glass-floored section that provides a thrilling experience and a unique perspective of Hanoi. Additionally, there are coffee shops where you can enjoy a classic Vietnamese coffee (more on that later) and some VR games that entertain the kids and add to the fun for everyone.

Hanoi Train Street

After visiting the Skydeck, checking into our room, and enjoying lunch at the hotel, we headed to Hanoi’s Train Street (HTS). Online, we found out there was a train scheduled at 3:30 PM. HTS is a unique and thrilling attraction where a narrow residential street comes alive as a train passes just inches away from homes and cafes. Lined with colorful buildings and bustling with local life, it’s a fascinating place to watch the train squeeze through while sipping Vietnamese elixirs (coffee, beer) and munching Vietnamese food from one of the cozy cafes. This blend of daily life and adventure makes it a must-see in Hanoi.

9:00 PM Train @ Hanoi Train Street

Unfortunately, we missed the 3:30 PM train due to traffic, and by the time we arrived, it had already passed. Determined not to miss the experience, we tried again on another night, and the photo we captured was of the nighttime train. The atmosphere was electrifying as the train approached—the warning bell rang, shopkeepers hurriedly moved their chairs indoors, guards cleared people off the tracks, and some even placed beer bottle caps on the rails to flatten them as the train sped by. When the train finally roared past, the crowd erupted in cheers and applause, awestruck by the thrilling spectacle.

If you’re wondering whether to visit this place during the day or at night, I’d highly recommend going at night! The glowing lanterns and the lively, international noisy crowd make it a truly magical experience. Be sure to check the train schedule in advance or ask the café owners for timings. One café owner told us there was a train at 8:30 PM, but the train we ended up seeing arrived at 9:00 PM. Initially, we thought it was a ploy to keep us there longer to eat, drink, and spend more, but it turned out to be worth the wait!

Lotte Aquarium

After visiting Hanoi Train Street (HTS), we headed to the Lotte Aquarium, having purchased a combined ticket for both the Skydeck and the aquarium. Located within the Lotte Center near West Lake, just a 20-minute drive from the Daewoo Hotel, this modern aquarium is an excellent destination for families. It offers an immersive experience showcasing diverse marine life, from vibrant coral reefs to mesmerizing jellyfish displays. A highlight for my child was participating in pearl harvesting, where she watched a pearl oyster being opened to reveal a pearl, which could then be crafted into jewelry. She also enjoyed the hands-on experience of touching marine creatures like shellfish, crabs, and starfish. The aquarium fosters a magical connection with the underwater world, making it a fun and educational outing for all ages!

The Dinner Disaster

The food in Vietnamese malls and hotels is predominantly non-vegetarian, with beef and pork being common ingredients. Both my wife and I stick to seafood and chicken but avoid red meat, while my daughter eats eggs and is just starting to experiment with chicken—though she won’t go near seafood or red meat. This made finding suitable food in the malls quite challenging, as most dishes contained beef or pork. In the end, we settled for ice cream instead.

Communication in Vietnam posed another challenge, as English is not widely spoken or understood. We quickly adapted by using Google Translate to communicate, a skill we relied on heavily. The girls were eager to try the iconic Vietnamese Pho, but given the limitations, we returned to our hotel and ordered chicken and vegetarian Pho instead as part of the dinner buffet. 80% of the buffet was out of reach!

Day 2: Ha Long Bay – Nature’s masterpiece

The day-long trip to Ha Long Bay began early, with a 7:30 AM meeting time at a designated point (for us, it was the Hanoi Opera House). A car transported us from our hotel to the meeting point, and punctuality was key—everyone adhered to the schedule, with a call from the driver/guide, 10 minutes in advance to ensure we were ready. From the meeting point, buses or limousines carried passengers to Halong Bay. At each stop, tour guides stepped off the bus to locate their assigned travelers, calling out names, scanning faces, and matching passengers with their groups—a lively process that felt like a cheerful treasure hunt.

The journey to Halong Bay shared with strangers (an international crowd from Japan, Taiwan, Mainland China, Australia, France, India, Germany, South Africa, and Jamaica), made for a lively and engaging travel experience. The tour guides, brimming with humor and charm, kept us entertained with amusing anecdotes, intriguing historical tidbits, and a detailed overview of the day’s itinerary. Their lively storytelling added a unique touch to the ride, making it both informative and enjoyable. However, when the tour guide took a break from narrating, the bus quickly fell into a peaceful silence as most passengers drifted off to sleep.

Cruise & Lunch

After a 2.5-hour drive, with a couple of planned and emergency pit stops, we arrived at Tuan Chau Harbor to board the boat and began our excursion exploring the stunning beauty of this UNESCO World Heritage Site. Onboard, we enjoyed a set menu lunch featuring a variety of Halong’s special dishes. Passengers were seated based on their dietary preferences, which ranged from vegan and vegetarian to non-vegetarian with specific preferences like eggs, beef, pork, or seafood. The guide carefully arranged seating to prevent conflicts, especially avoiding solo travelers claiming the best spots—a surprising number of whom were on the trip.

The food was excellent, and my wife and I opted for the seafood menu, which was a highlight for us. However, my daughter didn’t enjoy the vegetarian options and decided to rely on chips and chocolate for the rest of the day. Even that proved tricky, as many snacks in Vietnam are flavored with pork, beef, or seafood, making her scrutinize every label carefully before digging in.

TiTop Island

After lunch, we headed up to the deck to enjoy the stunning views of the islands and capture some photos. Our next stop was TiTop Island, where visitors could choose to swim or hike to the top for the best view. We opted for the trek, eager to take in the panoramic scenery. Each limestone island in Halong Bay is uniquely named based on its appearance, such as Chicken Island, adding to the charm of the experience. The hike to the top was steep, with over 500 steps, but the mesmerizing view made every step worth it. At the summit, we treated ourselves to refreshing coconut water, snapped some memorable pictures, and soaked in the breathtaking landscape before heading back down to rejoin the cruise.

Luan Cave

Our first stop after the hike was Luon Cave, located on Bo Hon Island. The area surrounding Luon Cave is rich with well-known attractions in Halong Bay, making it a hub of natural beauty. Directly in front of the cave is Turtle Island, while Heaven Gate lies to its right, adding to the picturesque setting.

Visitors have several options to explore Luon Cave: paddling through its waters by kayak, gliding on a traditional bamboo boat, or opting for a speedboat tour to see nearby attractions. We chose the speedboat, and it turned out to be an exhilarating experience. The driver skillfully maneuvered the boat, creating an adrenaline rush as we sped across the water. Along the way, our guide pointed out fascinating landmarks like Dragon Cave, Stone carved Lotus, and Mermaid, each adding a touch of wonder to the journey. It was a thrilling and memorable way to experience the beauty of Halong Bay.

Sung Sot Cave

Our next destination was the famous Sung Sot Cave, or the “Surprising Cave,” located on Bo Hon Island. As we approached, our tour guide shared fascinating details about the cave’s history and its unique features, preparing us for the experience. However, we were greeted by a long line of eager tourists—proof of the cave’s popularity as one of Halong Bay’s must-see attractions. Sung Sot Cave is a natural marvel, consisting of three chambers: a small entry chamber, a middle chamber, and an expansive, grand chamber that lives up to its name. Spanning a 2-kilometer walk, the cave is adorned with stunning stalagmites and stalactites that have been shaped over millennia by dripping water and time. The formations take on fascinating shapes and textures, sparking the imagination as you stroll through the illuminated path. The grand chamber, the highlight of the cave, is particularly breathtaking, with its vast space and dramatic formations. It’s no wonder the cave is named “Surprising”—its sheer scale and beauty leave visitors awestruck. Despite the crowd, the experience was well worth it, as we explored one of the most beautiful caves in Halong Bay.

Sunset

The return to the harbor was serene, a perfect end to our day at Halong Bay. As the sun set, the golden light reflected on emerald waters, casting a warm glow over the limestone islands. A gentle breeze carried a sense of tranquility, washing away urban stress. Standing on the deck, surrounded by nature’s beauty, we felt a profound peace—a magical moment that makes Halong Bay unforgettable.

Day 3: Ninh Binh Tour

The day began even earlier than Day 2, as we prepared to meet a new group of strangers at a different meeting point. Our guide, Mr. Luca, was a bit of a newcomer to managing a large bus tour, having mostly handled smaller limousine groups before. The itinerary for the day included a 110 km drive filled with diverse activities planned by the tour team: a visit to a historic temple, a cycling adventure, a serene boat ride, and a trek through stunning landscapes.

Dinh King Temple

The first stop was a temple. The Dinh King Temple is located in the ancient capital of Hoa Lu and was built in honor of Emperor Dinh Tien Hoang, who founded the Dinh Dynasty in the 10th century. The temple reflects the spiritual depth of Vietnam’s early dynasties. Intricate carvings of dragons, phoenixes, and mythical creatures adorn the temple, symbolizing strength, prosperity, and protection.

Vietnamese culture has deep-rooted connections to kingship and spirituality, with temples serving as both places of worship and symbols of reverence for past rulers. The five elements—metal, wood, water, fire, and earth—play a significant role in traditional Vietnamese cosmology, representing harmony between humanity and the natural world. The Vietnamese people are believed to embody characteristics of these elements based on their birth year. These elements shape compatibility in relationships, especially marriage. For example, water nourishes wood, making individuals of these elements a harmonious match, while fire and water may clash.

Cycling and Lunch

Our second stop was for a short cycling tour followed by lunch. Cycling was a fun way to work up an appetite while exploring the land like traditional Vietnamese locals. However, the cycle was a bit too large for my daughter, so she decided to piggyback while I pedaled through the scenic route. She did try it when there was no traffic! Along the way, there were plenty of photographers (paparazzi) ready to capture the moment, making us feel like celebrities!

Exploring the countryside on a bicycle offered a unique perspective, reminiscent of our cycling adventure in the paddy fields of Bali. Though this tour was brief, lasting about 30 minutes, we spent a memorable time winding through peaceful villages. The activity got our metabolism going, and we rewarded ourselves with a delicious Vietnamese lunch. A highlight of the meal was the fresh spring rolls—thin rice paper wraps filled with vegetarian or other tasty fillings. Healthy, light, and full of flavor, they were a perfect treat after the ride.

Trang An Grottoes

The afternoon was spent on a tranquil ride in traditional sampan boats, their unique design adding to the charm of the experience. These flat-bottomed wooden boats are rowed by local women (always talking and making jokes) and can accommodate up to four passengers. The ride felt like being part of a school of fish, effortlessly flowing together in perfect rhythm through the water. The cool weather enhanced the magic as we glided through the Trang An Caves, a mesmerizing network of interconnected caves and waterways surrounded by dramatic limestone mountains. However, caution was necessary—low cave ceilings meant you could easily bump your head if not careful!

Along the way, we stopped at ancient sites like Cao Son Temple and Suoi Tien Temple, nestled amidst lush greenery. These stops not only offered a chance to explore the spiritual history of the area but also served as a welcome restroom break during the two-hour journey.

Trang An’s ethereal beauty has also earned it international recognition as a filming location for Kong: Skull Island. Its otherworldly landscapes bring to life the feeling of stepping onto a movie set, blending stunning natural scenery, rich history, and a touch of cinematic magic.

Ngoa Long (Lying Dragon) mountain

The Ngoa Long Mountain, located in the Mua Cave area, offers one of the most stunning panoramic views in Ninh Binh. A popular trekking spot, it’s often said, “A trek a day keeps stress away!” Our guide cheerfully estimated 20 minutes to climb up, 20 minutes to soak in the views, and 20 minutes to descend—but the reality is never quite so balanced. The climb is steep, with two routes: a steeper left side boasting spectacular vistas and a slightly easier right side. We opted for the more challenging left route, navigating short, steep steps that tested our endurance.

Before tackling the ascent, we took a leisurely walk around the tranquil lotus ponds, capturing some beautiful photos. The climb was rewarding, and at the summit, the breathtaking view of rice fields, limestone karsts, and the surrounding landscape made every effort worthwhile. Ngoa Long Mountain is truly a trekker’s delight and a photographer’s paradise.

Day 4: Da Nang

We had an early morning flight to Da Nang (Flight VJ 503: Hanoi [HAN] to Da Nang [DAD], departing at 7:20 AM), which meant a hectic night of packing and an early scramble to the airport. Adding to the chaos, we had squeezed in a night visit to Hanoi Train Street—an experience we didn’t want to miss.

Sleep was scarce, coming in short bursts during the car ride, at the airport, and on the flight. Still, those quick naps were enough to recharge us for the day ahead. Our walking pace had shifted to “turtle mode,” and every step was a not-so-gentle reminder that our muscles were staging a protest after the Day 3 climb!

Once in Da Nang, we checked into Holiday Beach Resort, located on a beautiful stretch of My Khe beach. With no specific plans, we started the day with breakfast and a relaxing stroll along the shore. The waves were perfect for surfing, and we spent time watching skilled surfers and beginners alike tackling the high swells. We were not keen on jumping into the cold waters of the south china sea.

While soaking in the peaceful beach atmosphere, we spoke with our tour guide, who enthusiastically suggested visiting Hoi An. Her advice? “Just grab a taxi and go!” With no set plans, it felt like the perfect spontaneous next step.

Hoi An: Coconut Village

We visited the Hoi An Coconut Village and had an amazing time exploring its lush coconut groves. Riding in traditional round basket boats, expertly steered by local rowers (mostly men), we navigated through a maze of winding waterways. The guides entertained us with impressive boat tricks and even fished out crabs, adding to the excitement. The peaceful setting of the groves was beautifully complemented by the lively music and cheerful atmosphere. To make the experience even more special, we had a fun photo shoot, creating lasting memories of this vibrant slice of Vietnamese culture.

Hoi An: Old Town

After visiting the Coconut Village, we hopped into another taxi and headed to Hoi An Old Town. The charming streets welcomed us with their iconic yellow buildings and bustling riverside vibe. Since most shops in Hoi An only accept cash, our first stop was an ATM before we began exploring.

We wandered through the streets, passing boatkeepers along the river and inviting us for the 10:00 PM lantern-lighting rides. Along the way, we stopped for ice cream and enjoyed a delicious Vietnamese lunch at Miss Ly Café. The town seemed to understand Indian travelers well—the kind waitress went out of her way to offer true vegetarian dishes alongside seafood and chicken options, which we really appreciated.

Of course, we couldn’t resist picking up a few souvenirs, but the highlight for the kids (and us!) was the smoky ice cream, which felt like a show in itself. We also tried some street fruits—mango and jackfruit—despite our kids’ humorous warnings not to. To end the evening, we watched a lively dance performance by the riverside, soaking in the cultural charm of Hoi An. It was truly a day to remember!

Smoking ice cream in Hoi An comes surrounded by liquid nitrogen vapor, creating a dramatic “smoking” effect as you eat it. The cool, foggy spectacle makes it feel like more than just dessert—it’s an experience! It’s a must-try for anyone looking to add a little flair to their culinary adventures in Hoi An.

Day 5: Ba Na Hills

It was the last day of the year, and we started early, joining a group of fellow travelers. This group was primarily “Indian,” but not everyone was from India. In Da Nang, the convenience of buses picking you up directly from your hotel was a nice touch. The ride to Ba Na Hills was scenic, and soon we were on the cable car, ascending through misty clouds with breathtaking views of the Mo Stream below.

The English of our Vietnamese tour guide took some effort to understand, but with patience on both sides, we managed just fine. Like school kids, we were assigned group numbers and had to respond to roll calls, adding a touch of humor to the experience.

Ba Na Hills, originally developed in 1919 as a French hill station, has since transformed into a vibrant tourist destination. It felt like a mix of Disneyland and Ocean Park Hong Kong. Nestled atop the Trường Sơn Mountains, it offered a fascinating blend of natural beauty and man-made attractions. The cloudy, cold, and rainy weather added an ethereal charm, making it feel as if we were walking among the clouds. Thankfully, we had packed umbrellas to stay prepared.

Our adventure began with the iconic Golden Bridge, held aloft by the massive “Hands of God.” We arrived early enough to snap some photos before the crowds descended. Getting a clear shot, however, was nearly impossible with so many visitors vying for the same goal. Professional photographers stationed there had software to remove other people from your photos, but their services came at a steep price.

Expertly guided by our tour leader, we explored a variety of attractions, from the charming French Village to the peaceful 27-meter Buddha statue and the vibrant Flower Garden. Each stop felt like stepping into a different world, creating a truly diverse and exciting experience. We even took a tram ride to watch a 7D film, which transported us to fascinating places around the globe with its immersive visuals.

By lunchtime, my daughter was craving the comfort of a simple Indian meal. To help us beat the cold, we were treated to complimentary beers (truly elixirs from heaven). Our guide directed us to an Indian restaurant offering a buffet, which included plain dal and rice—just what my daughter wanted—alongside some pasta for variety.

After lunch, although we had some time left to explore Fantasy Park, the biting cold and windy weather (despite a forecast of 12°C, it felt much colder due to the rain and wind) forced us to seek warmth. We found ourselves, like many other families, huddled around a heater to warm our hands.

Ultimately, we decided to skip both the Fantasy Park and Linh Chua Linh Tu Temple. Instead, we opted to descend via the cable car and returned to the hotel by 5:30 PM, choosing comfort over braving the harsh weather in Ba Na Hills.

Happy New Year, 2025!

As the evening approached, we hoped the rain wouldn’t dampen our New Year’s plans—and luckily, it didn’t! We strolled around Da Nang, indulging in more Indian food for my daughter, sipping on unique Vietnamese coffees, getting a foot massage, and savoring fresh seafood, including a large crab. It was a perfect way to close out the year, blending familiar comforts with local flavors and experiences.

The streets were buzzing with life, packed with cafés serving every cuisine you can imagine, massage spas ranging from cozy to extravagant, and shopping marts eager to satisfy every whim of the visiting crowds. Every other corner seemed to have an Indian restaurant blaring ’90s Bollywood hits, adding a nostalgic desi touch to the lively atmosphere.

Big names like Sheraton and Marriott are racing to stake their claim along the beach, hinting at an even glitzier future for Da Nang. Meanwhile, the present was alive with loud party music echoing from various hotspots, each vying to pull in crowds for the midnight countdown.

Unfortunately, the famous 45-minute Da Nang fireworks show was canceled this year for environmental reasons, leaving some disappointed. But not all was lost—many families opted for a quieter celebration, strolling along the beach where small shops lit up the night with minor fireworks to keep the festive spirit alive. It was a vibrant mix of noise, nostalgia, and seaside serenity—a fitting end to the year!

We strolled back to our room, quietly responded to several New Year messages, and enjoyed a much-needed nap to prepare for another long (but last) day of travel.

Day 6: Hue

We reluctantly dragged ourselves out of bed for an early drive to Hue City. Hue is pronounced “h-way,” with an emphasis on the “h” sound at the start, followed by a drawn-out “u,” similar to saying “way.”

Once again, we were picked up and found ourselves among a group of sleepy international travelers, exchanging cheerful New Year wishes with everyone. Hue is renowned for its rich history and cultural significance, making it a must-visit destination.

The journey itself was an experience, taking us through the Hai Van Tunnel, the longest and most modern tunnel in Southeast Asia. This marvel of engineering not only made the drive smoother but also added a touch of awe to the trip.

Hue’s dialect is often considered more elegant and poetic compared to other regions in Vietnam. It’s said that the people of Hue speak with a soft, lilting tone that reflects the city’s historical ties to Vietnam’s royal past.

Lap An Lagoon

We made our first stop at Lap An Lagoon, located west of Lang Co Bay. This picturesque lagoon is surrounded by mountains and is home to several pearl farms. We spent some time taking in the serene views, sipping salt coffee, and watching local pearl farmers separating pearl oysters before continuing our journey.

Khai Dinh Tomb

Upon arriving in Hue, our first visit was to the Khai Dinh Tomb, one of the most beautiful and intricate royal tombs of the Nguyen Dynasty. The stunning architecture combines traditional Vietnamese and European elements, making it a truly unique historical site. The detailed carvings and mosaics were a testament to the grandeur of the Nguyen kings.

Many of the intricate carvings and mosaics in Khai Dinh Tomb were reportedly created by artists using their feet—a technique viewed as a bad omen in Vietnamese culture. There are two fascinating stories behind this unusual technique. One suggests that the artists used their feet as an act of protest against Emperor Khai Dinh, who raised taxes to fund the construction of the tomb. Another version claims the method was purely practical—ceiling paintings and carvings were done using feet so the artists could lie back and see the artwork as they worked, ensuring better precision and perspective.

Towering at the top of 127 steps, the tomb exudes grandeur. The structure is covered in blackened concrete, which gives it an imposing presence against the lush green backdrop of the mountains. The interior, however, bursts into vibrant life with colorful glass and porcelain mosaics adorning the walls. The most striking feature is the statue of Emperor Khai Dinh, cast in bronze and seated beneath a celestial ceiling painting that depicts a lively dragon amidst clouds.

Thien Mu Pagoda

Post-lunch (at a local restaurant), we visited the Thien Mu Pagoda, Hue’s oldest and most beautiful pagoda. This iconic site, perched on a hill overlooking the Perfume River, is a symbol of Hue’s spiritual heritage but also a serene retreat that offered a peaceful pause in our journey.

The name “Thien” means “heavenly” or “celestial,” while “Mu” translates to “lady,” together symbolizing a divine feminine spirit. According to legend, an old woman appeared on the hill where the pagoda now stands, prophesying that a great leader would build a temple here to bring peace and prosperity to the region. Inspired by this, Lord Nguyen Hoang ordered the construction of the pagoda in 1601, marking the beginning of its storied history.

The pagoda is also tied to a tragic love story in local folklore. Legend has it that couples who visit the pagoda together—especially those holding hands—might face separation afterward. While this remains a superstition, it adds an intriguing and mystical element to the otherwise serene and spiritual site. As our tour guide jokingly remarked, “If you’re looking for separation, just hold hands here!”

The Phuoc Duyen Tower, a seven-story pagoda, stands as an iconic symbol of Hue here. Each of its seven stories represents a step on the Buddha’s path to spiritual awakening. This concept is tied to the legend of Buddha’s seven steps at birth, where each step was marked by a blooming lotus flower, symbolizing purity and enlightenment.

There were many bonsai trees in the gardens here. Miniature trees, commonly known as bonsai, are referred to as cây cảnh or “ornamental trees” in Vietnamese culture. These living art forms are meticulously shaped to balance natural beauty and artistic vision, showcasing the grower’s skill and patience. Bonsai symbolizes harmony, balance, and interconnectedness, often representing nature’s grandeur in a small form. For many, caring for bonsai is a meditative practice fostering mindfulness and inner peace. In Vietnamese and East Asian traditions, they are associated with Feng Shui, believed to attract positive energy, prosperity, and good fortune. Historically, bonsai also signified wealth and status, often adorning the homes of nobles and scholars. Whether for artistic, spiritual, or cultural reasons, bonsai remains cherished worldwide.

Hue Citadel

Our last destination was the Hue Citadel, a UNESCO World Cultural Heritage site since 1993. This massive complex, with its walls, palaces, and gates, holds the legacy of the 13 Nguyen dynasty kings who ruled for over 140 years. Despite enduring the ravages of time and war, the Citadel continues to preserve its historical significance and charm.

Approximately 40% of Vietnamese people have the surname Nguyen, making it by far the most common surname in Vietnam. The prevalence of this surname is linked to the Nguyen Dynasty, the last imperial dynasty of Vietnam, which ruled from 1802 to 1945. During this time, many people adopted the surname as a sign of loyalty or to align themselves with the ruling class.

The undisputed highlight of the day? Our royal visit to the royal toilet! Who knew history could be so flush with grandeur?

After a two-hour walk exploring Hue’s rich history and culture, exhaustion caught up with us, and we quickly fell asleep on the bus ride back to Da Nang. The driver smoothly navigated the route, and we arrived at our hotel by 6:30 PM, marking the official end of our holiday adventures.

Last Evening

To celebrate our final evening in Vietnam, we indulged in a feast of fresh seafood by the beach, including a mouthwatering lobster, and savored one last round of aromatic Vietnamese coffee—a fitting farewell to the culinary delights of this vibrant country.

Special Mention: Vietnamese Coffee

Vietnamese coffee culture is incredibly diverse, with a wide variety of options to explore. Popular choices include Cà Phê Sữa Đá (iced coffee with milk), Cà Phê Đen (black coffee), Cà Phê Trứng (egg coffee), Cà Phê Dừa (coconut coffee), Bạc Xỉu, and the luxurious Cà Phê Chồn (weasel coffee). Another unique offering is Avocado Coffee, a creative blend of flavors. Each type reflects the rich coffee heritage of Vietnam and offers something for every palate. You can have coffee hot or iced!

Done Done

If you’ve reached here, congratulations! This travelogue was penned down immediately after our trip, before the busy workdays started rolling in and the memories began to blur.

One thing that still makes us laugh is how I kept calling “Da Nang” “Ha Dang”, completely forgetting the real name. The names of the places we visited—Ha Noi, Hoi An, Ninh Binh, Ha Long, Da Nang, Ba Na, Hue, Sung Sot, Bo Hon, Luon, Trang An, Cao Son, Suoi Tien, Ngoa Long, Thien Mu, and Lap An—felt like tongue twisters at times. I couldn’t help but mix up Da Nang with Ha Dang, much to everyone’s amusement! My daughter made sure I was properly “punished” for my slip-up with endless giggles. We googled Ha Dang, and found that to be a real person’s name!

I suggest you put Vietnam on your bucket list! Cheers

Embracing Uncertainty

Picture a bamboo grove in a storm. While mighty oaks crack and fall, bamboo stems bow to the ground yet spring back unbroken. This simple truth about flexibility defeating rigidity lies at the heart of handling uncertainty. Today’s world spins faster than ever – markets transform overnight, technologies reshape entire industries, and a single event can send ripples across the global economy.

Ancient Wisdom

“The mind is very restless, turbulent, strong and obstinate, O Krishna. It appears to me that it is more difficult to control than the wind.” – Arjuna

When facing uncertainty, ancient texts offer surprising clarity. The Bhagavad Gita speaks of focusing on actions rather than outcomes – karma (choices, actions, causation) over karmaphala (fruit, outcomes). This isn’t just philosophy; it’s practical advice for today’s leaders. When you focus solely on results, fear can paralyze you. When you focus on taking the right actions, clarity emerges.

The Dance of Adaptation

Leadership is a dance with uncertainty. Sometimes you lead, sometimes you follow, but always you must stay in motion. The ancient concept of dharma teaches us to act with integrity without clinging to specific outcomes.

“In a gentle way, you can shake the world.” – Mahatma Gandhi

Like bamboo, successful organizations build resilience through flexibility. They maintain strong roots – their core values and principles – while adapting their strategies to changing winds. This balance between steadfastness and adaptability creates lasting success.

Seeing through Maya’s Veil

In Sanskrit philosophy, Maya doesn’t simply mean illusion (a reality distortion field). Think of it like watching shadows on a cave wall – what we see is real, but it’s not the complete truth. In leadership, this concept is surprisingly practical.

The Upanishads teach about “neti neti” – Sanskrit for “neither this, nor that” – a powerful method for finding truth by eliminating what isn’t true. Instead of immediately grasping at solutions when facing uncertainty, we can systematically eliminate what we know isn’t working or true.

“When you have eliminated the impossible, whatever remains, however improbable, must be the truth.” – Sherlock Holmes

Today’s leaders can practice this by asking deeper questions (why, when, what, which, where, who): What assumptions are we making? What aspects of this situation am I misleading? Which assumptions need challenging? What alternatives haven’t we considered? What possibilities lie beyond our current view? etc.

When we stop clinging to our first interpretations and systematically examine our assumptions, new opportunities often emerge from unexpected directions.

“The important thing is not to stop questioning. Curiosity has its own reason for existence” – Albert Einstein

The Empty Cup

An old Zen story tells of a master pouring tea into a student’s already full cup until it overflows. The lesson? We cannot receive new insights when our minds are already full of preconceptions. In times of uncertainty, we must first empty our cups – let go of what we think we know to discover what might be.

Final Thoughts

Uncertainty isn’t the leader’s enemy – it’s the space where innovation happens. Like bamboo in the storm, true strength lies not in standing rigid against change, but in dancing with it. By embracing uncertainty while staying true to our principles, we transform challenges into opportunities for growth.

Leaders who thrive in uncertainty don’t try to control the uncontrollable. Instead, they build resilient organizations that bend but don’t break, adapt but don’t compromise their values, and turn change into opportunity. This way of leading isn’t just about survival – it’s about finding grace in chaos and strength in flexibility.

“In the middle of difficulty lies opportunity.” – Albert Einstein

WHY THIS BLOG?

I was talking to leaders taking their year-end breaks, and I sensed the weight of uncertainty (organization changes, inability to make lasting changes, frequent pivots, and idea conflicts) wearing them down. Hence, this blog.

While I don’t have a perfect solution, some things have helped me in my journey: Taking action instead of freezing. Staying true to your values. Being flexible in your plans but firm in your integrity. Trusting your team. Listening and Respecting Ideas. Keeping your mind healthy – not letting it trap you in victim thinking and impossible scenarios. This last one (keeping the mind healthy) is/was difficult for me to practice sometimes, but being in these cycles multiple times is helping me/has helped me.

Leadership isn’t about having answers to everything. It’s about staying grounded while keeping flexible.

Generative AI in Healthcare

Today’s popular ChatBots have evolved to the point where they can ‘search’ the internet to build a profile of a person. While the idea of profiling might seem unsettling, what’s even more fascinating is their ability to dynamically integrate with external tools. This means these bots don’t just acquire context—they actively use it to take meaningful actions. And that’s a game-changer. In industries like healthcare, where inaction and delays can have serious consequences, this capability to bridge information and execution is nothing short of revolutionary.

Insights Are Necessary But Not Sufficient

We can design a world-class computer vision algorithm to detect potholes on roads—no doubt about that. The real value of such technology lies in its ability to prevent small potholes from escalating into traffic nightmares. But here’s the catch: insight without action is as good as scrap metal. Knowing there’s a pothole on Road A is useful, but if it’s not fixed, that insight is wasted. This is where Generative AI steps in, not just to detect problems but to close the loop by orchestrating solutions. Imagine a team of AI agents—a detector, a work approver, a fixer, and even a finance approver—all collaborating to decide which potholes get priority. Given limited budgets and resources, the challenge becomes an optimization problem: which roads to repair to maximize safety, improve traffic flow, and minimize frustration. With high volumes of insights, this kind of collaborative decision-making can cut through the ‘insights fatigue’ and turn knowledge into action.

Healthcare faces a similar dilemma. There’s a heavy emphasis on monitoring—producing insights that are supposed to drive action—but the gap between knowing and doing often remains wide. Take the case of a diabetic patient struggling to lower their HbA1c from 8.5. The problem might not be awareness but action: lifestyle changes like regular walks, strength training, better diet choices, or even adhering to medication schedules. The truth is, most people know what to do; the challenge lies in execution.

This is where a hyper-personalized approach becomes critical—like having a health coach who not only reminds you to do your sit-ups but makes sure you squeeze in 10 while you’re still on the call. Habits, once formed, become second nature (think brushing your teeth). But doctors don’t have the bandwidth to coach every patient. Enter Generative AI, which can act as a virtual co-pilot for healthcare professionals. Imagine a digital assistant that mirrors a doctor’s conversational style while incorporating a coach’s motivational touch. This AI can identify when a patient is straying off course, focus discussions on actionable lifestyle changes, and tackle one problem at a time. If medication compliance isn’t the issue, it can hone in on diet or exercise—whatever the patient needs most. Generative AI brings the promise to move insights to actions.

Generative AI Use Case Sampler

Generative AI has the potential to revolutionize healthcare by enabling smarter, more personalized, and action-oriented solutions. By leveraging diverse modalities like text, images, audio, and video, this technology can bridge gaps in patient care, education, and operational efficiency. Here are some potential use cases:

Text-to-Text: Personalized Discharge Summaries
Generative AI can generate hyper-personalized discharge summaries tailored to individual patients. By pulling data from EMRs and provider-recommended templates, these summaries can be presented in clear, actionable language, preferably in the patient’s preferred language. From follow-up instructions to medication schedules, this empowers patients to take control of their recovery with confidence.

Image-to-Text: Radiology Reports Made Smarter
Deep learning has already enabled precise interpretation of radiology images. Generative AI can take this further by generating highly personalized reports for different audiences, such as PCPs, patients, or specialists. By reducing radiologists’ workload and improving turnaround times, it can ensure quicker and more efficient delivery of critical insights. Moreover, Generative AI can allow radiologists to query images for specific details or compare similar images by vectorizing (creating embeddings) the image and reports.

Text-to-Image: Visualizing Patient Education
Generative AI can help patients better understand their diagnoses and treatment options by creating personalized visualizations. A picture truly is worth a thousand words, especially in healthcare, where complex concepts can be challenging to convey through text or speech alone.

Image-to-Image: AI-Powered Image Reconstruction
Generative AI can enhance the quality of medical images through advanced reconstruction techniques, improving both speed and resolution. This capability can boost diagnostic accuracy and provide healthcare professionals with the clarity needed for more informed decisions.

Text-to-Speech: Accessibility for All
For patients with disabilities or language barriers, text-to-speech technology can make medical information more accessible. Whether through audio outputs or Braille conversion, Generative AI ensures inclusivity in patient communication and care delivery.

Audio-to-Text: Seamless Medical Transcription
Generative AI can convert conversations—between doctors and patients or among clinicians—into structured medical records. This technology streamlines documentation, allowing healthcare professionals to focus on patient care while simultaneously generating structured, actionable records.

Text-to-Video: Transforming Medical Education
Generative AI can transform training and education by generating personalized, easy-to-follow video content. From simplifying complex medical topics to creating interactive learning experiences, it offers a more engaging way to educate both patients and healthcare professionals.

Text-2-Text: Generative AI

A Text-2-Text generator can be used to build a story to recite to a child to help her develop better study habits. With a simple prompt, Generative AI (courtesy: ChatGPT) can craft a compelling story—like one inspired by Harry Potter—that captures the child’s imagination while embedding useful lessons. For example, a magical artifact like a “Focus Fuser” might serve as the centerpiece of the story, teaching readers how small, consistent actions can lead to meaningful progress.

Now shift to a healthcare setting. Medical jargon, while essential for precision, often confuses patients. Generative AI can take complex radiology reports and translate them into clear, patient-friendly language. For example, instead of saying, “Impression: The findings are consistent with a moderate right-sided pleural effusion,” the AI might explain: “The results of your imaging test show a moderate amount of fluid in the space around your right lung, known as a pleural effusion. There’s no sign of pneumonia or a collapsed lung. Your doctor may recommend further tests to get more clarity if needed.”

This translation doesn’t just simplify language—it empowers patients to understand their health better, enabling them to make informed decisions.

Text-2-Image: Generative AI

A Text-2-Image Generator can recreate familiar concepts with stunning precision and creativity. Take the classic representation of the “Three Wise Monkeys”—each embodying the principles of “see no evil, hear no evil, speak no evil.” With a simple prompt, AI can generate a vivid image of three monkeys, each covering their eyes, ears, or mouth, perfectly capturing the symbolic meaning behind the concept.

This same technology demonstrates its utility in healthcare through advanced imaging capabilities. For instance, it can generate a synthetic frontal chest X-ray (CXR) image of a patient with specific features, such as simple pleural effusion and focal airspace opacity. This isn’t just an exercise in realism—it supports medical education, research, and even diagnostic validation. The ability to generate medically accurate images empowers healthcare professionals and researchers to innovate and enhance patient care. It also supports the cause of patient privacy.

Image-2-Text: Generative AI

The image-to-text generator capability is a game-changer for accessibility, enriching experiences for visually impaired individuals and enabling deeper engagement with visual content through natural language. Auto-generation of key features from an image can enable search (E.g., get me pictures with a blue lake and snow-capped mountains).

On the other hand, in a healthcare radiology context, Generative AI can generate detailed radiology reports (E.g. CXR 2 views – AP/LAT) such as: “The heart size and mediastinal contours are normal. There is no evidence of focal airspace consolidation, pleural effusion, or pneumothorax. No acute bony abnormalities are observed, although degenerative changes are present in the thoracic spine.”. This not only reduces the workload on radiologists but also enhances accuracy and efficiency in delivering diagnostic insights. By automating the creation of such reports, AI enables healthcare professionals to focus more on patient care while ensuring timely and precise interpretations.

Reasoning is Complex

(*) The above reasoning is not to show any tool or technology in a Bad light.

Reasoning can be a tricky beast, as you’ve probably noticed by now. While a human might have gone for answers like, ‘Maybe the male prisoner was released!’ or ‘Plot twist—he escaped!’, AI takes a different approach. It sticks to the context you give it, like a dog chasing a stick. But here’s the catch: if you want AI to keep doing its mind-blowing, jaw-dropping, ‘WOW’ job, you’ve got to toss it the right stick—clear context and solid rules. Otherwise, you might end up with answers as wild as a soap opera plot!

Closing Quote

‘The machine does not think, but it reveals our thinking.’

Generative AI quality is proportional to the quality of the information and context we give it—whether it’s clear or confusing. This means the responsibility isn’t just on the AI (or its builders) but also on us (the users) to ensure it is used in a way that reflects important values. When used wisely, it can help us improve care, build stronger connections, and create a fairer world for everyone.

2024: Emerging Technology Trends

This blog is my attempt to make sense of the transformative technology trends shaping 2024, organizing them into a structure that helps me—and hopefully you—grasp their impact. From sweeping macro shifts to granular micro innovations, I’ve distilled my observations and reflections into this post to explore how these trends are reshaping our world. My goal is to spark ideas and inspire curiosity as we navigate the ever-evolving frontier of technology.

From Star Trek Dreams to Today’s Realities

As a lifelong fan of Star Trek, I often find myself marveling at how much of its futuristic vision has seamlessly blended into our everyday lives. What was once the realm of science fiction is now science fact, and it’s astonishing to see how the imaginative worlds of Gene Roddenberry have inspired generations of innovators. Let me take you on a journey through some of these Star Trek dreams and their counterparts in today’s technology:

Communicators → Smartphones: Captain Kirk’s communicator is today’s smartphone, letting us call, navigate, shop, and even control our homes.
PADDs → Tablets: Starfleet’s sleek devices are now tablets like iPads and Kindles, offering portable, powerful access to information.
Universal Translators → Google Translate: Star Trek’s universal translator lives on in Google Translate, breaking language barriers in real-time.
Tricorders → Portable Medical Scanners: Dr. McCoy’s tricorder inspired tools like GE’s VScan, revolutionizing portable medical diagnostics.
Replicators → 3D Printing: Captain Picard’s replicator echoes in 3D printing, creating tools, prosthetics, and more layer by layer.
Holodeck → Virtual and Augmented Reality: The Holodeck is here with VR and AR, immersing us in gaming, collaboration, and virtual experiences.
Voice-Activated Computers → Siri and Alexa: Starfleet’s voice-activated computers are now Siri, Alexa, and Google Assistant, responding to commands daily.
Data → AI like ChatGPT: The android Data foresaw today’s AI, like ChatGPT (+Robotics), transforming creativity, problem-solving, and workflows.
Memory Banks → Cloud Computing and Big Data: Starfleet’s vast knowledge banks mirror today’s cloud computing and data lakes, offering limitless storage.

Star Trek’s genius lies not just in its speculative tech but in how it inspires us to push the boundaries of the possible. We may not have cloaking devices, or the Prime Directive fully figured out yet, but the trajectory is clear—we’re steadily turning sci-fi into sci-reality. The question is no longer if but when.

A Quote from a modern-day philosopher and visionary

“Imagine a digital product as the spirit animal of a human, guiding and empowering her in her journey. The question isn’t just how we maintain these companions, but how we nurture them to thrive alongside us in the pursuit of seamlessness.”

— CHATGPT (Prompt Engineered by Nitin Mallya)

Every post needs a thought-provoking quote, right? But in the age of AI, why not ask ChatGPT, our modern-day philosopher? A little prompt engineering led to a quote that resonated with a Star Trek twist.

Remember Chakotay from Star Trek: Voyager and his guiding spirit animal? It was more than mystical—it was a companion, helping him navigate challenges. Today, AI is becoming just that: a copilot, not just a tool, but a partner empowering us to explore and solve problems.

Another Quote

There’s a quote I love: “A product/service that gets better after we sell it.” I’m not sure who first said it, but it perfectly captures the magic of today’s connected technologies. It’s not just about creating products; it’s about creating experiences that evolve and improve over time.

For a salesperson, this quote highlights the thrill of the sell. For a product manager or engineer, it’s all about delivering the better. But the real brilliance lies in the platform/ecosystem—the invisible engine that keeps adding value long after the sale is made.

In the world of hardware, this might mean something tangible, like adding a dashcam to your car. But in software, the possibilities are limitless. Think of apps that refine workflows, tools that make complex tasks effortless, or AI systems that transform product usage. The game has shifted; it’s about enhancing every moment of the user’s experience over the product’s entire lifecycle.

Quantum Digitalization: The Journey to Hyper-Personalization

The trajectory of technology often feels like a dance between the micro and the macro, the tangible and the intangible. I call this journey Quantum Digitalization—a fusion of precision and expansiveness that is reshaping how we interact, innovate, and solve problems. It’s not just a trend; it’s a profound transformation unfolding in stages, each one moving us closer to a world where hyper-personalization isn’t merely possible—it’s inevitable.

I found inspiration in the AI paper, “Attention is All You Need,” and I’ve borrowed its clarity to describe the stages of this evolution. Each stage has a theme and a guiding principle that frames the technology and its impact.

Digitize: Data Is All You Need

The first step in this journey is to Digitize, or as we might say in India, Digify. This stage is about making sense of chaos—bringing structure to the unstructured and transforming information into a usable form. It’s the shift from paper to pixels, where workflows are streamlined, knowledge is centralized, and insights become attainable. However, the magic of digitization isn’t in collecting more data; it’s in collecting the right data. Too much can lead to a data swamp—overwhelming and unusable. Too little creates gaps that even the smartest algorithms cannot bridge. This stage is the foundation—the roots that support the tree of innovation. Without a strong base, the rest of the journey is shaky at best.

Digitalize: Experience Is All You Need

Once the data is in place, the next step is to Digitalize. This stage is often misunderstood. It’s not about Digitizing; it’s about creating meaningful, engaging experiences. Technology shifts here from being a tool to becoming a partner, enhancing how people interact with systems and processes. This stage is about connection—devices talking to each other, systems understanding users, and interactions that feel intuitive. Imagine seamless experiences, where diagnostics, decisions, and delivery happen almost invisibly. It’s no longer just about solving problems; it’s about delighting users. Unfortunately, many still confuse digitalization with digitization or equate it solely with software. But digitalization is much more—it’s about creating ecosystems where experience becomes the driver of adoption and innovation.

Qubitize: Accuracy is all you need

As the journey advances, we step into the stage of Qubitize, where the guiding theme is Accuracy is all you need. This is the realm of quantum technologies, which bring a level of precision, speed, and security that traditional systems simply cannot match. It’s here that the boundaries of what is possible begin to dissolve, unlocking a future once confined to science fiction.

At the heart of Qubitize are three transformative domains: quantum sensing, quantum computing, and quantum communications. Each represents a leap forward, reshaping how we measure, compute, and interact in ways that redefine accuracy.

Quantum Sensing is revolutionizing how we perceive the physical world. Traditional sensors—no matter how advanced—have limitations in precision and sensitivity. Quantum sensing taps into phenomena like quantum superposition and entanglement to deliver ultra-precise measurements of time, magnetic fields, gravity, and more. For example, quantum-enabled imaging devices could detect changes at the molecular level, opening doors to breakthroughs in fields like healthcare, geophysics, and environmental monitoring. Imagine medical scans with resolutions so high that they can detect diseases at their earliest stages or environmental sensors that can map underground resources with pinpoint accuracy. This level of sensing not only enhances accuracy but also expands what we can observe and understand.

Quantum Computing takes problem-solving to a whole new dimension. Unlike classical computing, where bits are either 0 or 1, quantum bits—or qubits—exist in multiple states simultaneously, enabling computations at unprecedented speeds. This allows quantum computers to solve complex problems, like optimization or molecular modeling, that would take classical systems centuries to process. For instance, quantum algorithms can revolutionize logistics by finding the most efficient routes in real time, or they can accelerate drug discovery by simulating molecular interactions at a scale and speed previously unimaginable. Quantum computing isn’t just faster; it’s smarter, enabling us to approach problems in ways that were previously inconceivable.

Quantum Communication offers a paradigm shift in data security and transfer. At its core is Quantum Key Distribution (QKD), which uses the principles of quantum mechanics to create encryption keys that are theoretically unbreakable. By leveraging entangled particles, QKD ensures that any attempt to intercept or tamper with the transmission is immediately detectable, making it a cornerstone for ultra-secure communications. Beyond security, quantum communication networks are laying the groundwork for a quantum internet—an interconnected web of quantum devices that could transform how information is shared globally. This leap will redefine the concept of trust in digital systems, safeguarding sensitive information in ways that traditional cryptographic methods cannot match.

Qigitalize: Seamlessness Is All You Need

Finally, we arrive at Qigitalize—the frontier where the physical and digital worlds blur into one seamless continuum. This stage is about fluidity, where boundaries dissolve, and technology adapts to us rather than the other way around. Imagine ecosystems where robots autonomously handle tasks, collaborating with humans only when necessary. Picture a world where virtual and real merge effortlessly, enabling interactions that are frictionless and immersive. Collaboration transcends barriers, and hyper-personalization becomes a given. It’s a vision of a world that feels intuitive, connected, and limitless.

AI: A Microtrend Shaping the Future

AI is in the midst of an explosion—a rapid evolution transforming how we build, use, and interact with intelligent systems. Let’s dive into how AI is expanding its influence in four transformative dimensions: learning styles, tasks, data modalities, and system designs.

Learning Styles: How AI Masters the World

AI’s ability to learn has evolved dramatically. It began with supervised learning, where labeled data taught machines to distinguish between cats and dogs. While groundbreaking, this method relied heavily on expensive, annotated datasets. Then came unsupervised learning, revealing patterns in unlabeled data, like customer segmentation. Semi-supervised learning bridged the gap, blending smaller labeled datasets with large unlabeled ones for efficiency. Reinforcement learning marked another leap, enabling AI to learn through trial and error by interacting with environments—think Chess or Go. But the game-changer is self-supervised learning, where AI generates its own labels from data structures, powering massive models like GPT. Combine that with transfer learning, which reuses pre-trained models for new tasks, and federated learning, which keeps data private while training across decentralized systems, and you see AI’s growing adaptability. These advances are turning AI into an increasingly self-reliant, scalable force.

Learning Tasks: From Assistants to Creators

AI has transcended its origins as an assistant. Initially focused on classification (e.g., “Is this a cat?”) and regression (e.g., “What will sales be next quarter?”), AI has now become a creator. Generative AI systems can write essays, design artwork, or even produce videos from simple prompts. This creative power is revolutionizing industries, from marketing and entertainment to education and healthcare. But generative AI is just the beginning. Multimodal capabilities allow AI to integrate text, images, and audio seamlessly. Imagine an AI that can describe a picture in words, generate a matching soundscape, or create immersive VR experiences—all from a single prompt. This evolution isn’t just about automating tasks; it’s about transforming AI into a collaborator, pushing the boundaries of creativity and innovation.

Data Modalities: Beyond Text/Images

As AI grows, it’s venturing into new data territories. Vision and language remain the bedrock, powering breakthroughs in computer vision, natural language processing, and search engines. AI’s ability to interpret haptics—touch and pressure data—is revolutionizing robotics and prosthetics, enabling machines to “feel.” Meanwhile, spectral data, which captures wavelengths beyond the visible spectrum, is transforming agriculture, defense, and medical imaging. These new modalities are extending AI’s reach into uncharted territories, creating once unimaginable possibilities.

AI Systems: From Assistance to Autonomy

AI systems are evolving at a pace that’s reshaping the boundaries of what machines can do. What’s most fascinating is how our trust in these systems has grown, allowing them to take on tasks that were once the sole domain of humans. This evolution is about more than efficiency—it’s about unlocking entirely new possibilities.

Let’s start with semi-autonomous systems. These are like reliable coworkers who need just a little oversight. Take drones, for example. They can map terrain, deliver packages, and navigate through complex environments on their own. Yet, for critical decisions—like rerouting in extreme weather or avoiding restricted airspace—they defer to human judgment. This “autonomy with a safety net” allows machines to handle repetitive or high-risk tasks while keeping us in the loop for truly important calls.

Then, there are multi-agent systems, which take teamwork to a whole new level. Imagine a city where AI agents manage traffic lights, not individually, but as a coordinated network. These systems communicate in real-time, adjusting signals across intersections to ensure the smoothest flow of vehicles. The result? Reduced congestion, lower emissions, and happier commuters. It’s like having an army of super-intelligent traffic controllers working in perfect harmony. But multi-agent systems don’t stop at collaboration—they can also compete. In financial markets, trading bots operate as independent agents, analyzing trends, predicting outcomes, and executing trades faster than any human could. Some even collaborate to manipulate market conditions for mutual gain, demonstrating just how dynamic and adaptive these systems can be.

The next step is fully autonomous systems. These are the true independents, capable of sensing, deciding, and acting entirely on their own. Self-driving cars are a prime example: they don’t just follow a programmed route—they interpret traffic signs, anticipate pedestrian movements, and adjust to road conditions in real-time. Industrial robots are similarly autonomous, assembling products, identifying defects, and even scheduling their own maintenance. But here’s where the challenge lies: when we hand over complete control to machines, the stakes are higher than ever. Safety protocols must be flawless, and decision-making processes must be transparent and accountable. We’re making great strides, but full autonomy still demands rigorous refinement.

And now, for the most exciting part? economic participation. AI is no longer just about doing what we already do, only faster or better—it’s about exploring new frontiers. Take generative AI systems in intraday trading. These systems analyze vast amounts of market data, identify patterns, and execute strategies faster than any human trader. What’s even more remarkable is their ability to adapt in real-time, learning from market fluctuations and refining their tactics on the fly. These agents aren’t just assistants; they’re active participants in the economy, creating wealth independently.

Beyond finance, autonomous systems are venturing into creative and strategic territories. Imagine an AI agent tasked with designing marketing campaigns. It could analyze audience data, draft ad copy, create visuals, and even determine the optimal time to launch—all without human intervention. Or consider healthcare, where AI agents might coordinate treatments for patients, consulting with doctors only for the most complex cases. These examples aren’t just theoretical—they’re already in motion, signaling a profound shift in how we think about machine intelligence.

This evolution is about more than technology—it’s about redefining what machines can do for us. AI is no longer confined to automation or assistance. It’s becoming a collaborator, a decision-maker, and even an innovator in its own right. And here’s the kicker: we’re only scratching the surface of what’s possible. As these systems grow more capable, adaptive, and independent, they’ll continue to push the boundaries of intelligence, creativity, and impact. This is the dawn of a new era, where machines don’t just follow—they lead. And the journey has only just begun.

How can this be regulated!?

Balancing Act of AI Regulation

Regulating AI is a delicate balancing act—a tightrope walk between fostering innovation and ensuring safety. Governing bodies are tasked with a daunting challenge: to embrace the transformative potential of AI while protecting society from its unintended consequences. It starts with risk classification, determining which systems are harmless tools and which might pose significant risks, like surveillance technologies with the potential for misuse. Coupled with this is the need for transparency and explainability—because if even the creators can’t fully understand their models, how can users or regulators trust them?

Data privacy is another cornerstone of responsible AI regulation. With AI systems handling sensitive information, particularly in fields like healthcare, strict safeguards are essential to prevent misuse and protect personal data. Beyond that, regulators must confront harmful applications head-on, banning systems that perpetuate discrimination or exploitation, such as social scoring mechanisms. These measures aren’t about stifling progress; they’re about building a foundation of trust, ensuring that AI not only thrives responsibly but also paves the way for sustainable growth. By navigating this tightrope, we can unlock AI’s full potential while keeping its promises aligned with human values.

Summary

Technology in 2024 is transforming at an unprecedented pace, bringing us closer to the Star Trek dream of seamless human-machine collaboration.

AI micro trend is pushing boundaries, evolving from simple assistants to autonomous decision-makers. Multi-agent systems optimize cities, while generative AI creates solutions in real time, transitioning from insights to actions. Machines are no longer just tools—they’re collaborators, turning science fiction into reality and revolutionizing how we work, live, and connect.

The Quantum Digitalization macro trend is all about redefining hyper-personalization, moving from digitizing workflows to creating immersive, Qigitalized ecosystems where the physical and digital worlds blend effortlessly. Powered by quantum sensing, computing, and communication, this trend is driving unprecedented accuracy, security, and innovation.

“Trend is fashionable. Fashions change. Old trends come back. But when they return, they’re never quite the same—they carry the weight of the past, reimagined for the present. A trend reborn isn’t just a repeat; it’s a remix, blending nostalgia with innovation, and reminding us that in the cycle of change, the new is often just the old with better timing. What will trend next?”

Kashmir

It’s Tvisha’s summer vacation, and due to work pressures, we could not take her out of Bangalore in April. So she was bugging us to get out of Bangalore. After comparing Leh/Ladakh, Sikkim, Andamans, and Kashmir, we finally settled on Kashmir. I had been to Kashmir when I was very young and had fond memories of the place. I was unsure whether it was safe to visit Kashmir, and after much online research and talking to the travel agents, we zeroed in on Kashmir.

Some friends were going on a Bangalore-Leh drive (9K+ Kms), and others were going to monasteries in Sikkim. We had recently been to monasteries in Bhutan, and the girls were not in the mood for another long drive! In December 2021, we enjoyed a seven-day long drive (Bangalore-Coonoor-Kotagiri-Coimbatore-Mahabalipuram-Bangalore). The weatherman predicted rains in Andamans, and the daughter wanted to make Olaf! So, Kashmir was inevitable!

After comparing rates and dates with our favorite travel agent (SOTC), we finally booked our trip with the MakeMyTrip holidays.

Day 1 – Bangalore to Srinagar

There is a direct Indigo Flight (6E 797) from Bangalore to Srinagar that starts in Bangalore at a convenient time (~9:15 AM) and reaches Srinagar post noon (~2:00 PM) with a brief halt at Amritsar.

When the plane landed at the Srinagar airport, all passengers were in awe of Srinagar’s beautiful hills (some capped with snow). At the airport, our driver greeted us, and he instantly realized that we South Indians were in Kashmir to see snow in summer! He promised us ample snow in Sonmarg and Gulmarg, making Tvisha very happy!

It was a short drive from the airport to our first hotel in Srinagar. It was not a classic hotel but a houseboat. The houseboat we stayed in was Naaz Kashmir (https://www.naazkashmir.com/) and was located in Nageen Lake (and not Dal Lake). All the lakes in Srinagar are connected, but Nageen Lake is less commercial and crowded.

The rest of the day was free for us to enjoy the houseboat! We spent our time taking photos, watching fishermen and birds catch fish, eating pakoras, listening to the music of the water and birds, listening to the prayers from different surrounding Mosques, dressing up in a traditional dress, and a short shikara ride.

It was an abrupt stop to the fast life we are used to, staring back at nature and adoring its beauty. We talked to each other as a family and were not immersed in our gadgets! However, Tvisha was excited to show off her first day in Kashmir to her friends in a WhatsApp call!

Naaz Kashmir served us well – candlelight dinners, food per our needs, and recommendations about Shikara rides. So we took their advice and decided to take the 4:30 AM four hours Shikara ride from Nageen Lake to Dal Lake.

Staying in a houseboat is like sharing a house with other equally clueless and excited families.

Day 2 – Srinagar (Shikara Ride)

We kept the alarm to wake us up at 4:00 AM to prepare for our 4:30 AM shikara ride. We woke up to the alarm and morning prayers at the lake. It reminded me of my pre-college days when 4:00 AM had become a routine for studies. The caretakers were already up and knocked on our doors to ensure that we were ready and sent us on the shikara with some hot Kashmiri tea (Kahwa) and snacks.

It was bitter cold (for us) and pleasant for the locals. We tugged ourselves into the blanket available in the shikara. Our little braveheart sandwiched herself conveniently between her parents, refused to step out of the blanket throughout the ride, enjoyed the frequent warm rubs, and did not hesitate to nap. The shikara moved slowly, confidently, and thoughtfully through the lake.

The cold air on the face and the calming sound of the shikara moving in the lake is an unforgettable experience. Most of the locals and birds were up at 4:30 AM!! As we rowed through the lake, we could see the homes of the locals – men and women at work. Everyone that met the eye returned a welcoming smile.

We spotted a water snake, various colored water lilies, and birds like eagles, geese, mallards, pochards, gadwalls, pintails, waders, coots, and the common teal. The shikara rider helped us identify the birds.

We could see water vapor causing fog in some places due to the difference in water temperatures and the surroundings.

The shikara rider explained to us in detail the unique farming done by the locals in the lake, i.e., the process in which they grow vegetables. These farmers grow carrots, radishes, turnips, and other vegetables in the soil that floats on flora beds.

Such floating gardens are maintained all over the lake, and the farmers move through their plots on boats carrying their harvest to the lake’s market area. The rider took us through the floating gardens and the lotus farms to the floating vegetable market on dal lake. We bought some flowers and seeds and had some hot Kahwa (Kashmiri Tea) and snacks packed for us by the Naaz Kashmir at this market.

The rider then took us to the dal lake (pronounced as दल and not दाल), and at this time of the day, it was empty except for the locals fishing out the water weeds to compost them for use in their farms. It was a beautiful sunrise to watch at the dal lake, wade through the markets (Meena Bazaar), and spot traditional homes. The ride back was slow, and the gentle swaying motion of the shikara ride put us to sleep for a few minutes.

Day 2 – Sonmarg (सोनमर्ग and not सोनमIर्ग)

At around 9:30 AM, after breakfast, we started driving to Sonmarg, a hill station in the Ganderbal district.

It was about two hours drive. The roads were not too good, and there were traffic jams in a few places. If we had left about an hour earlier, we might have saved about 1/2 hour of jam time. It was visitor traffic. However, after the early morning Shikara ride, we needed some time to fuel and freshen ourselves before our next adventure.

As we reached closer to Sonmarg, we could see the snow-capped mountains, feel the drop in temperature, and breathe the superior quality air. Sonmarg was much cooler than Srinagar but fortunately pleasant (even for our visitor skin).

During the drive, we could see streams of water – the tributaries of Jhelum – Lidder, Sind, and Neelum. The sight of white water forcing itself down the mountains was strangely peaceful. It reminded us of our stay in Salt Lake City, Utah.

The driver told us that during the winter months, heavy snowfall blocks NH-1. So a tunnel is being built to keep the road open year-round. We found trekkers ready to trek from Sonmarg to Leh (“the rooftop of the world”) and groups wanting to drive through the Zoji La pass. Our driver informed us that driving through the Zoji La pass is a must-do off-road adventure. However, he recommended we not take our city car and take a four-wheel drive as the roads are narrow and rocky. It seems it’s a trek worth doing! So, now this one is also added to the backlog!

However, our goal for today was relatively simple. We either trek or take ponies to the Thajiwas glacier. This glacier is a favorite summer destination at Sonmarg.

My daughter is always excited and happy around animals. We were not sure whether we (“The Adults”) needed ponies; however, the agents there convinced us that the pony ride is a “must-do” in Sonmarg. We gave in; however, it would have been a simple to moderate walk/trek and less burden on the animals in hindsight. The ponies were expensive, about 2750 INR / person (we negotiated out more than 50% of the original demand. Better negotiators got it at around 1500 INR / person). We also got a photographer for us to be more hands-free. The pony ride was uphill, downhill, and through cold water streams.

The scenery was picture perfect, a silvery scene set against green meadows and a clear blue sky. The views were captivating, and we outsourced the picture capture activity to our photographer and instead enjoyed the views. The air was too fresh to keep our masks on.

At the glacier, Tvisha finally made her Olaf! She was throwing snowballs at us in all directions, even when we were haggling with the locals to reduce the sledding costs! She was able to go up and down the snow and was soaking in the happiness, unlike us. Then, after an uphill trek in the snow, we sat down on a rock and came down sledding. The weather was not too cold, and if it were not for the time to return to Srinagar and limited food options, we would have spent more time up at the glacier.

We encountered the local police (to the surprise of the locals), who helped us reduce the cost of sledding from 3500/- to 500/- INR (though we ended up paying 1000 INR per person for sledding). It’s funny that the locals keep saying (क्या आप खुश है, हमे भी खुश कीजिये) “Are you happy? Make us happy!”, a method to get more money from the visitors. These people make money only during the summer months (visitor months) and try to make the most of it. COVID & CONFLICT closures have been hard on them. We found these people to be cheerful, happy, and helpful. So, we did not hesitate to give (tips) more than we thought was reasonable!

We stopped at a roadside restaurant to have some delicious chole-puri and dal-makhani for Tvisha on our way back. We reached back around 7:00 PM for another candlelight dinner at Naaz Kashmir. The owners of Naaz Kashmir had moved our luggage out of the room as they thought that we booked for only one night, and they realized that there was a communication issue between MakeMyTrip and their reservation team. They made up for it by giving us a superior room, a chocolate cake for Tvisha, and several apologies.

We took a warm bath and crashed soon after!

We were woken up by Tvisha mid-night as she started vomiting and was feeling very sick! She had no temperature but did not look well. We were not sure whether it was the altitude, change in food/water, or a stomach bug. Thankfully, we had carried medicines – “Enterogermina” and “Calpol” in our first aid kit!

Day 3 – Pahalgam

We started around 9:45 AM after thanking the caretakers for their service at Naaz Kashmir and completing the check-out formalities. Tvisha was sick and only managed to eat mangoes for breakfast. The drive to Pahalgam was long but on more convenient and motorable roads. We wanted to stop for Kahwa and visit apple farms, but Tvisha would only sleep in the car. So, we drove straight to our hotel, the Radisson Golf Resort.

The Pahalgam scenery was unique, with white water streams and a backdrop of pine trees and snow-capped mountains. The temperature was pleasant and leaning towards too cool.

All we could do on day-3 was check in and rest. Tvisha slept all afternoon and night. She was running a slight fever, so we consulted our family doctor, and she diagnosed her to have the stomach flu. So we requested the hotel to get us medicines (the medicine shop was ~2KMS far), and they helped us.

The trip managers advised us to visit Abu Valley, Betaab Valley, Chandanwari, and Baisaran valley (Mini-Switzerland). However, we had lost 1/2 a day and had to check out by 1:00 PM the next day (Day-4). So we decided that if Tvisha feels better on Day-4, we will do the Baisaran valley.

Pahalgam (First Village) gets its name from Hindi “पहला गांव” and Shiva devotees visit this place frequently in summer to pilgrimage to the Amarnath Cave for the Darshan of the only ice stalagmite Shiva Linga. The pilgrimage usually starts from Chandanwari but was closed due to road work during our travel.

Day 4 – Pahalgam

Tvisha woke up ready for her next adventure; however, I felt queasy in my stomach. It was my turn to fall sick. However, Dolo came to my rescue, and after an insignificant breakfast, we all jumped up on horses to visit the Baisaran valley. This trek is captivating, and it’s better to trek than take horses. However, a just recovered Tvisha and a queasy daddy were not in any state to trek. So, horses again!

Baisaran Valley is a hilltop green meadow dotted with dense pine forests and surrounded by snowcapped mountains. This famous offbeat tourist place is excellent for those wanting to spend a quiet time in the company of nature. It also serves as a campsite for trekkers going to Tulian Lake. Some of the famous tourist points you can see en route to Baisaran are Pahalgam Old Village, Kashmir Valley Point, and Deon Valley Point. You can also enjoy panoramic sights of Pahalgam town & Lidder Valley from here.

We returned to our hotel after about four hours (~1:00 PM) ride to Baisaran. We completed our checkout and had a delicious rainbow trout tandoori (a local delicacy) for lunch. The driver told us that butter or olive fry is better. After a sumptuous meal, we departed by car to Gulmarg.

Day 4 – Pahalgam to Gulmarg (गुलमर्ग )

It took about 3.5 hours from Pahalgam to reach Gulmarg. Gulmarg is a ski destination and is famous for winter sports. It is called the meadow of flowers in summers. Our driver informed us that they have to use chains on wheels in winter, and the snowfall in Gulmarg is heavy. The beauty of Gulmarg was different (in a good sense) than Pahalgam and Sonmarg. The first thing that strikes out is the lush green meadows (in summers).

We camped in the Hilltop Hotel. At first glance, the hotel seems jaded, faded, and under maintenance. While that is true, the rooms were well designed, and the in-room dining service was good. The service at this hotel was better than any of our previous experiences in Kashmir, even though they were ok at room cleaning services, breakfast variety, bathroom accessories, changing towels, or responding to your hails. We were satisfied that we could have a warm bath, eat something edible, and reach on time for the gondola ride the following day. The hotel is close to the gondola ride and ice skating rink.

Dolo carried me only this far, and I had slight chills and crashed for the night. I hoped, wished, and prayed that the fever would help kill the virus/bacteria (stomach bug) for me to enjoy Gulmarg the next day.

Day 5 – Gulmarg – Gondola Ride to Kongdoori

The chills were gone in the morning; however, I was still queasy. So, I trusted my best friend (Dolo) and braved the gondola.

MakeMyTrip was able to arrange gondola tickets for the first stage (Base Station to Kongdoori). The tour guide told us that the second stage tickets (Kongdoori to Apharwat) were unavailable (sold out). However, in hindsight, we did not regret not being able to do the second stage.

The gondola wait lines and wait times are infamous. People start queuing at 7:00 AM, even though the ride opens at 9:30 AM. We reached the queue at around 8:00 AM and found our tour guide. The people standing in the line entertained themselves by fighting with others who tried to move ahead. Words and punches were flying until the ride opened at 9:30 AM. The locals were also amused at the sight. The trip was about 10 minutes, and the wait time to board the ride was 2 hours. Dolo kept me on my feet.

The gondola ride is short, and the views are terrific. The valley is picturesque, and we can spot the snowcapped Himalayas, Apharwat, and Mud houses from the ride. We could also see the unlucky visitors (those who could not get a gondola ticket) trekking or using horses to climb up to Kongdoori. This trek is a moderate to challenging hike.

At Kongdoori, people had to queue again to ride to Apharwat, and the queue was equally long. We were happy that we don’t have to wait in another queue.

Gondola Ride to Kongdoori (Friend: Dolo)

At Kongdoori, the horse owners were bugging us to take a horse ride to the waterfall. However, we discussed it with our tour guide and decided to trek. Trekking (and not horse riding) was the best decision. We could stop to look at the multi-colored flowers in the meadows, jump over streams, trek with the goats, stop to hear the sound of silence, spot lizards, take photos, and experience rocky trails.

When we reached the waterfall, we had to trek in snow to get up to the mouth of the waterfalls. The snow trek was challenging.

However, the mountain water was delicious and pure. We drank from the waterfall, and this water tasted better than any mineral or filtered water. So we filled a bottle to quench our thirst for the return journey.

If I travel again to destinations like these, I will remind myself to buy shoes with some grip.

Finally, we sledded downhill and relished on some delicious maggie cooked in the mountains waters. Strangely, maggie gave us the strength to trek back to the gondola ride station.

We took a different route to see the valley and meadows from a different perspective. This route was shorter and required us to climb uphill and roll downhill.

Again, the view was picture perfect!

After reaching the base station, we rushed to feed ourselves a late lunch. The lunch was good. We did not have any energy left to do the ATV rides and decided to skip them and relax in the room. Then, in the evening, we went down to the cafeteria to have some snacks. Our legs could not tolerate any more walking and would only walk back in the direction of the hotel room. So, we snuggled back into the room and watched the evening walkers from the comfort of the room.

Days are long in Gulmarg, and it’s bright even at 7:00 P.M.

I recovered from the stomach bug (Thanks! Enterogermina), and now it was time for my wife to fall sick to the same bug! She had a better immune reaction to the bug than my daughter or me; however, she sought help from Dolo and Enterogermina to fight off the bug.

Day 6 – Back to Srinagar

Gulmarg is about ~50Kms from Srinagar. So, the return journey was short. We woke up late and lazy, and left for Srinagar after a late breakfast.

We stopped to see apple farms and drink delicious green apple juice. We tasted various homemade pickles and bought lotus stem pickles from the farmer. Lotus stem (कमल-ककड़ी), locally known as “Nadru” is grown in shallow parts of water bodies like ponds and lakes and is a vastly enjoyed ingredient in Kashmiri cuisine.

We also stopped to have some premium Kahwa and buy some dry fruits (Walnuts) and condiments (Kesar).

We checked into Radisson Srinagar, the first hotel in Kashmir, where we found women employees. My wife had a heart-to-heart talk about women empowerment with the ladies there!

After a good lunch, we headed to see the oldest temple in Kashmir, the Shankaracharya Temple, dedicated to Lord Shiva. The temple is a monument of national importance and is protected by ASI (Archeological Society of India). There are many steps to climb, and the view of Kashmir valley from the hilltop is superb. It was very windy up the hill and pleasant. Unfortunately, cameras were not allowed for a picture remembrance. After blessings from Lord Shiva, we decided to stroll the gardens of Srinagar.

Unfortunately, due to the holiday rush and this day being a Sunday (many locals were out sightseeing), we saw only the Botanical Garden and the Chashme Shahi. We missed the Tulips as the Tulip Garden was closed a few days back (Peak season: April). The botanical garden was a nice walk, and the flowers in Chashme Shahi were exquisite. My daughter enjoyed taking photos of several flowers.

We had earlier decided to ride the Shikara again at Dal Lake; however, we decided to dart back to the hotel, looking at the rush and weather. Finally, we finished the day with a lavish buffet dinner.

Day 7 – Back to Home @ Bangalore

The only eventful activity was the security checks at the Srinagar Airport. We must step out of our cars at least a kilometer before the airport and have to get ourselves, the car, and the bags checked.

We left Srinagar entirely mesmerized by the beauty of Kashmir, and I decided to pen this down in a blog (for us) so that this never fades from (our) memory. So this blog is my first travel blog.

After a few more doses of Enterogermina and home food, my wife got better. We have returned to our workaholic ways and keep discussing our Kashmir trip with friends and family.

Data Descriptors (Stats, Relations, Patterns)

Data analysts look for descriptors in data to generate insights.

For a Data aggregator, descriptive attributes of data like size, speed, heterogeneity, lineage, provenance, and usefulness are essential to decide the storage infrastructure scale, data life cycle, and data quality. These aggregator-oriented descriptions are black-box perspectives.

For a Data analyst, descriptive statistics, patterns, and relationships are essential to generate actionable insights. These analyst-oriented descriptions are white-box perspectives. The analysts then use inferential methods to test various hypotheses.

Descriptive Statistics

Data analysts usually work with a significant sample of homogenous records to statistically analyze features. The typical descriptive statistics are – measures of location, measures of center, measures of skewness, and measures of spread.

E.g., A 23 member cricket team of three different states has players of the following ages:

Karnataka: [19,19,20,20,20,21,21,21,21,22,22,22,22,22,22,23,23,23,23,24,24,24,25,25]

Kerala: [19,19,20,20,20,21,21,21,22,22,22,22,23,23,23,23,23,24,24,24,24,24,24]

Maharashtra: [19,19,19,19,19,19,20,20,20,20,20,21,21,21,21,22,22,22,23,23,24,24,25]

Numbers represented this way does not help us detect patterns or explain the data. So, it’s typical to see the tabular distribution view:

AGE	Karnataka	Kerala	Maharashtra
19	2	2	6
20	3	3	5
21	4	3	4
22	5	4	3
23	4	5	2
24	3	6	2
25	2	0	1

Age Distribution of State Players

This distribution view is better. So, we would like to see measures of center for this data. These are usually – MEAN, MEDIAN, and MODE.

MEAN is the average (Sum Total / # Total)
MEDIAN is the middle number
MODE is the highest frequency number

Measure	Karnataka	Kerala	Maharashtra
MEAN	22	22.1	21
MEDIAN	22	22	21
MODE	22	24	19

Measures of Center

This description is much better. So, we would like to see this graphically to understand the skewness.

The age distribution is symmetrical for Karnataka, skewed to the left for Kerala, and skewed to the right for Maharashtra. The data analyst may infer that Karnataka prefers a good mix of ages, Kerala prefers player experience, and Maharashtra prefers the young.

The data analyst may also be interested in standard deviation, i.e., a measure of spread. The standard deviation symbol is sigma (σ) for a sample and is the MEAN distance from the mean value of all values in the sample. Since a distance can be positive or negative, the distance is squared, and the result is square-rooted.

Measure	Karnataka	Kerala	Maharashtra
Standard Deviation	1.8	1.7	1.8

Measure of Spread

In our example, a measure of location (quartiles, percentiles) is also of interest to the data analyst.

Percentile	Karnataka	Kerala	Maharashtra
25 Percentile	21	21	19.5
50 Percentile	22	22	21
75 Percentile	23	23.5	22
100 Percentile	25	24	25

Measure of Location

The table above shows that the 50 percentile value is the median, and the 100 percentile is the maximum value. This location measure is helpful if the values were scores (like in an exam).

Combining statistics and display to explain the data is the art of descriptive statistics. There are several statistics beyond the ones described in this short blog post that could be useful for data analysis.

Time-series Data Patterns

The time-series data has trends, variations and noise.

A trend is the general direction (up, down, flat) in data over time.
Cyclicity variation is the cyclic peaks and troughs in data over time.
Seasonality variation is the periodic predictability of a peak/trough in data.
Noise is meaningless information in data.

The diagrams below provide a visual explanation:

“Ice cream sales are trending upward,” claims an excited ice-cream salesman.

“Used Car value is trending downward,” warns the car salesman

“Every business has up and down cycles, but my business is trending upwards,” states a businessman.

“It’s the end of the month, so, Salary and EMI season in user accounts, so the transaction volume will be high,” claims the banker.

“There is some white noise in the data,” declared the data scientist.

Data Relationships

Data analysts seek to understand relationships between different features in a data set using statistical regression analysis.

There could be a causal (cause and effect) relationship or simply a correlation. This relationship analysis helps to build predictors.

A simple measure of linear relationship is the correlation coefficient. The measure is not relevant for non-linear relationships. Correlation coefficient of two variables x and y is is calculates as:

correlation(x, y) = covariance(x, y) / (std-dev(x) * std-dev(y))

It’s a number that in the range [-1,1]. Any number closer to zero implies no correlation, and closer to either extremity means higher linear correlation.

Negative one (-1) means negatively linearly correlated
Positive one (1) means positively linearly correlated
Zero (0) means no correlation

Example: Let’s take this random sample.

X	Y	Y1	Y2	Y3
1	3	-3	83	-100
2	8	-8	108	250
3	15	-15	146	-50
4	24	-24	98	150
5	35	-35	231	-50
6	48	-48	220	155
7	63	-63	170	-125
8	80	-80	100	-150
9	99	-99	228	-12
10	120	-120	234	190

Sample Data

X and Y	X and Y1	X and Y2	X and Y3
1	-1	0.6	0

Correlation coefficient

Visually, we can see that as X increases, Y increases linearly, and Y1 decreases linearly. Hence, the correlation coefficient is positive (1) and negative (-1), respectively. There is no linear relation between X and Y3, and hence, the correlation is 0. The relationship between X and Y2 is somewhere in between with a positive correlation coefficient.

If X is the number of hours bowler practices and Y2 is the number of wickets, then the correlation between the two can be considered positive.

If X is the number of hours bowler practices and Y3 is the audience popularity score, then the correlation between the two can be considered negligible.

If X is the number of years a leader leads a nation, and Y or Y1 is his popularity index, then the correlation between the two can be considered linearly increasing or decreasing, respectively.

Summary

Data analysts analyze data to generate insights. Insights could be about understanding the past or using the past to predict the (near) future. Using statistics and visualization, the data analysts describe the data and find relationships and patterns. These are then used to tell the story or take actions informed by data.

V’s of Data

Volume, Velocity, Variety, Veracity, Value, Variability, Visibility, Visualization, Volatility, Viability

What are the 3C’s of Leadership? “Competence, Commitment, and Character,” said the wise.

What are the 3C’s of Thinking? “Critical, Creative, and Collaborative,” said the wise.

What are the 3C’s of Marketing? “Customer, Competitors, and Company,” said the wise.

What are the 3C’s of Managing Team Performance? “Cultivate, Calibrate, and Celebrate,” said the wise.

What are the 3C’s of Data? “Consistency, Correctness, and Completeness,” said the wise; “Clean, Current, and Compliant,” said the more intelligent; “Clear, Complete, and Connected,” said the smartest.

“Depends,” said the Architect. Technologists describe data properties in the context of use. Gartner coined the 3V’s – Volume, Velocity, and Variety to create hype around BIG Data. These V’s have grown in volume 🙂

5V’s: Volume, Velocity, Variety, Veracity, and Value
7 V’s: Volume, Velocity, Variety, Veracity, Value, Visualization, and Visibility

This ‘V’ model seems like blind men describing an elephant. A humble engineer uses better words to describe data properties.

Volume: Multi-Dimensional, Size

“Volume” is typically understood in three dimensions. Data is multi-dimensional and stored as bytes—a disk volume stores data of all sizes. Data does not have volume! It has dimensions and size.

A person’s record may include age, weight, height, eye color, and other dimensions. The size of the record may be 24 bytes. When a BILLION person records are stored, the size is 24 BILLION bytes.

Velocity: Speed, Motion

Engineers understand the term velocity as a vector and not a scalar.

A heart rate monitor may generate data at different speeds, e.g., 82 beats per minute. I can’t say my heart rate is 82 beats per minute to the northwest. Hence, heart rate is a speed. It’s not heart velocity. I can say that a car is traveling 35 kilometers per hour to the northwest. The velocity of the vehicle is 35KMPH NW.

Data does not have direction; hence it does not have velocity. Data in motion has speed.

Variety: Heterogeneity

The word variety is used to describe differences in an object type, e.g., egg curry varieties, pancake varieties, sofa varieties, tv varieties, image data format varieties (jpg, jpeg, bmp), and data structure varieties (structured, unstructured, semi-structured). Data variety is abstract and is a marketecture term.

Heterogeneity is preferred because it explicitly states that:

Data has types (E.g., String, Integer, Float, Boolean)
Composite types are created by composing other data types (E.g., A Person Type)
Composite types could be structured, unstructured, or semi-structured (E.g., A Person Type is semi-structured as the person’s address is a String type)
Collections contain the same or different data types.
Types, Composition, and Collections apply to all data (BIG or not).

Veracity: Lineage, Provenance

Veracity means Accurate, Precise, and Truthfulness.

Let’s say that a weighing scale reports the weight of a person as 81.5 KG. Is this accurate? Is the weighing scale calibrated? If the same person measures her weight on another weighing scale, the reported weight might be 81.45 KG. The truth may be 81.455 KG.

Data represent facts, and when new facts are available, the truth may change. Data cannot be truthful; it’s just facts. Meaning or truthfulness is derived using a method.

Lineage and provenance meta-data about Data enables engineers’ to decorate the fact with other useful facts:
1. Primary Source of Data
2. Users or Systems that contributed to Data
3. Date and Time of Data collection
4. Data creation method
5. Data collection method

Value: Useful

If Data is a bunch of facts, how can it be valuable? Understandably, the information generated from data by analyzing the facts is valuable. Data (facts) can either be useful to create valuable information or useless and discarded. We associate a cost to a brick and a value to a house. Data is like bricks used to build valuable information/knowledge.

Summary

I did not go into every V, but you get the drill. If an interviewer asks you about 5V’s in an interview, I request you to give the standard marketecture answer for their sanity. The engineer’s vocabulary is not universal; technical journals publish articles in the sales/marketing vocabulary. As engineers/architects, we have to remember the fundamental descriptive properties of data so that the marketecture vocabulary does not fool us. However, we have to internalize the marketecture vocabulary and be internally consistent with engineering principles.

It’s not a surprise that Gartner invented the hype cycle.

Data Aggregation (Map, Filter, Reduce)

Data engineers think in batches!

Thinking in batches reminds me of a famous childhood story.

Once upon a time, a long, long time ago, there was a kind and gentle King. He ruled people beyond the horizon, and his subjects loved him.

One day, a tired-looking village man came to the King and said, “Dear King, help us. I am from a village beyond the horizon. It’s been raining for several days. My village chief asked me to fetch help from you before disaster strikes. It took me five days to walk to the Kingdom, and I am tired but glad that I could deliver this message to you.”

“I am glad that you came for help. I will send Suppandi, my loyal Chief of Defence, to assess the damage and then send help,” said the King. “Suppandi, you have your orders. Now, go. Assess the damage, report to me, and help,” ordered the King.

Suppandi left to the village beyond the horizon on his fastest horse. When he reached the town, the town was flooded, and Suppandi felt the urge to return to the King quickly to inform him about the floods. So, he drove his horse faster and reached the Kingdom in 1/2 day. He went to the King and told him. “Dear King, the village is flooded. I went in a day and came back in 1/2 day to give you this information.”

Suppandi was pleased with himself. However, the King wanted more information. “Suppandi, please tell me whether people in the village have food, are children hurt? What can we do more to help?”

“I will find out, Dear King,” said Suppandi. He left again on his fastest horse. This time he reached in 1/2 day. He figured that people don’t have food, and many children are hurt and homeless. He raced back to the Kingdom. “Dear King, I reached in 1/2 day and came back in another 1/2. The villagers don’t have food to eat, and they are hungry. Several children are hurt and need medical attention,” said Suppandi.

This time the King had more questions. “Dear Suppandi, what did the village chief say? What can we do for him?”

“Dear King, I will find out. Let me leave to the village immediately,” said Suppandi.

Chanakya was eagerly listening in to the conversation. He told Suppandi, “Dear Suppandi, you must be tired. Let me take over. Take some rest.”

Immediately, Chanakya ordered his men to collect food, water, clothes, medicines, and doctors. He asked for the fastest horses, and along with several men and doctors, he left for the village beyond the horizon. When he reached, the town was flooded, and people were on their home terraces. He found several houses destroyed and hungry kids taking shelter under the trees, and many wounded villagers.

He ordered his men to save the villagers skirting the flood, protect all children, feed them, and take them to a safe place. He also called the doctors to attend to the wounds.

The men built a temporary home outside the village to give shelter to the homeless. They waited for a few days for the rain and flood to subside. When it was bright and sunny, Chanakya, his men, and the villagers cleaned the village, re-built the homes, and deposited enough food and grains for six months before saying goodbye.

Chanakya reached the Kingdom and immediately reported to the King. The King was anxious. He said, “Chanakya, you were gone for two weeks with no message from you. I was worried. Did you speak to the village Chief?”

“Dear King, Yes, on your behalf, I spoke to the village chief. I found that the village was flooded, so we rescued all the villagers, attended to the wounded, fed them, re-built their homes, and left food and grains for six months. The people have lost their belongings in flood, but all of them are safe, and they have sent their wishes and blessings for your timely help,” said Chanakya.

The King was pleased. “Chanakya, I should have sent you earlier. You are a batch thinker! Thank you,” said the King.

Suppandi was disappointed. He had worked hard to drive to the village and report to the King as instructed, but Chanakya gets all the praises. To this date, he still does not understand and is hurt.

Most non-data engineers are like Suppandi; they use programming constructs like “for,” “if,” “while,” and “do” on remote data. Most data engineers are like Chanakya; they use the programming constructs like “map,” “filter,” “reduce,” and “forEach.” Programming with data is always functional/declarative, while traditional programming is imperative.

There is nothing wrong with acting like Suppandi; he is the Chief of Defence. But, some cases require Chanakya thinking. In architectural language, Suppandi actions move data to algorithms, and Chanakya actions move algorithms to data. The latter works better when there is a distance and cost-to-travel between data and algorithms.

This difference in thinking is why data engineers use SQL, and traditional engineers use C#/Java. SQL uses declarative commands that are sent to the database to pipeline a set of actions on data. The conventional programming languages have caught up to the declarative programming paradigm by supporting lambda functions (arrow functions), and map/filter/reduce style functions on data collections. The map/filter/reduce style functions allow compilers/interpreters to leverage the underlying parallel compute backbone (the expensive eight-core CPU) or use a set of inexpensive machines for parallel computing. They are abstracting away parallelism from the programmer. The programmer helps the compiler/interpreter to identify speed-improvement opportunities by explicitly programming declaratively.

Mapping

Instead of iterating over a collection one at a time, a map is a function to apply another function to all elements of a collection. The map function may split the collection into parts to distribute to different cores/machines. The underlying collection remains immutable. In general, mapping could mean one-2-one, one-2-many, and many-2-one; and is the process of applying a relation (function) to map an element in the domain with an element in the range. In the case of computing, mapping does not change the size of the collection.

E.g., [1,2,-1,-2] => [1,4,1,4] using the squared relation is a many-2-one mapping

var numbers = [1, 2, -1, -2];
var x = numbers.map(x => x ** 2);
console.log(x);

[1,4,1,4]

E.g., [1,2,-1,-2] => [2,3,0,-1] using the plus one relation is a one-2-one mapping

var numbers = [1, 2, -1, -2];
var x = numbers.map(x => x + 1);
console.log(x);

[2, 3, 0, -1]

E.g., [1,2,-1,-2] using the plus one and squared relation is a one-2-many mapping

var numbers = [1, 2, -1, -2];
var x = numbers.map(x => [x + 1, x ** 2]);
console.log(x);

[[2, 1], [3, 4], [0, 1], [-1, 4]]

E.g., An SQL Example of a one-2-one mapping

SELECT Upper(ContactName)
FROM Customers

MARIA ANDERS
ANA TRUJILLO
ANTONIO MORENO
THOMAS HARDY

Filtering

Instead of iterating over a collection one at a time, a filter is a function to return a subset of elements that match criteria. The filter function may split the collection into parts to distribute to different cores/machines. The underlying collection remains immutable. Examples:

var numbers = [1, 2, -1, -2];
var x = numbers.filter(x => x > 0);
console.log(x);

[1, 2]

SELECT *
FROM Customers
WHERE Country="USA"

Number of Records: 13

CustomerID	CustomerName	ContactName	Address	City	PostalCode	Country
32	Great Lakes Food Market	Howard Snyder	2732 Baker Blvd.	Eugene	97403	USA
36	Hungry Coyote Import Store	Yoshi Latimer	City Center Plaza 516 Main St.	Elgin	97827	USA
43	Lazy K Kountry Store	John Steel	12 Orchestra Terrace	Walla Walla	99362	USA
45	Let’s Stop N Shop	Jaime Yorres	87 Polk St. Suite 5	San Francisco	94117	USA

Reduce

Instead of iterating over a collection one at a time, a reduce is a function to return a single value. The reduce function may split the collection into parts to distribute to different cores/machines. The underlying collection remains immutable. Examples:

var numbers = [1, 2, -1, -2];
var x = numbers.reduce((sum,x) => sum + x, 0);
console.log(x);

SELECT count(*)
FROM Customers

Number of Records: 1
count(*)
91

Pipelining

When multiple actions need to be performed on the data then it’s a norm to pipeline the actions. Examples:

var numbers = [1, 2, -1, -2];
var x = numbers
  .map(x => x + 1) //[2,3,0,-1]
  .filter(x => x > 0) //[2,3]
  .map(x => x ** 2) //[4,9]
  .reduce((sum, x) => sum + x, 0) //13
console.log(x);

SELECT Country, Upper(Country), count(*)
FROM Customers
WHERE Country LIKE "A%"        
GROUP BY Country

Number of Records: 2

Country	Upper(Country)	count(*)
Argentina	ARGENTINA	3
Austria	AUSTRIA	2

Takeaway

Data Engineers use Chanakya thinking to get work done in batches. Even streaming data is processed in mini-batches (windows). Actions on data are pipelined and expressed declaratively. The underlying compiler/interpreter abstracts away parallel computing (single device, multiple devices) from the programmer.

Think in Batches for Data.

Data Quality (Dirty vs. Clean)

Data Quality has a grayscale, and data quality engineers can continually improve data quality. Continual quality improvement is a process to achieve data quality excellence.

Dirty data may refer to several things: Redundant, Incomplete, Inaccurate, Inconsistent, Missing Lineage, Non-analyzable, and Insecure.

Redundant: A Person’s address data may be redundant across data sources. So, the collection of data from these multiple data sources will result in duplicates.
Incomplete: A Person’s address record may not have Pin Code (Zip Code) information. There could also be cases where the data may be structurally complete but semantically incomplete.
Inaccurate: A Person’s address record may have the wrong city and state combination (E.g., [City: Mumbai, State: Karnataka], [City: Salt Lake City, State: California])
Inconsistent: A Person’s middle name in one record is different from the middle name in another record. Inconsistency happens due to redundancy.
Missing Lineage (and Provenance): A Person’s address record may not reflect the current address as the user may not have updated it. It’s an issue of freshness.
Non-analyzable: A Person’s email record may be encrypted.
Insecure: A Person’s bank account number is available but not accessible due to privacy regulations.

The opposite of Dirty is Clean. Cleansing data is the art of correcting data after it is collected. Commonly used techniques are enrichment, de-duplication, validation, meta-information capture, and imputation.

Enrichment is a mitigation technique for incomplete data. A data engineer enriches a person’s address record by adding country information by mapping the (city, state) tuple to a country.
De-Duplication is a mitigation technique for redundant data. The data system identifies and drops duplicates using data identities. Inconsistencies caused by redundancies require use-case-specific mitigations.
Validation is a mitigation technique that applies domain rules to verify correctness. An email address can be verified for syntactical correctness by using a regular expression (\A[\w!#$%&’+/=?{|}~^-]+(?:\.[\w!#$%&'*+/=?{|}~^-]+)@(?:[A-Z0-9-]+.)+[A-Z]{2,6}\Z). Data may be accepted or rejected based on validations.
Lineage and Provenance capture is a mitigation technique for data where source or freshness is critical. An image grouping application will require meta-data about an image series (video) collected like phone type and captured date.
Imputation is a mitigation technique for incomplete data (data with information gaps due to poor collection techniques). A heartrate time-series data may be dirty with missing data in minutes 1 and 12. Using data with holes may lead to failures, so a data imputation may use the previous or next value to fill the gap.

These are cleansing techniques to reduce data dirtiness after data is collected. However, data dirtiness originates at creation time, collection time, and correction time. So, a data cleansing process may not always result in non-dirty data.

A great way to start with data quality is to describe the attributes of good quality data and related measures. Once we have a description of good quality data, incrementally/iteratively use techniques like CAPA (corrective action, preventive action) with a continual quality improvement process. Once we are confident about data quality given current measures, the data engineer can introduce new KPIs or set new targets for existing ones.

Example: A research study requires collecting stroke imaging data. A description of quality attributes would be:

Data Quality Attribute	Description
Data Lineage & Provenance	– Countries: {India} – Imaging Types: {CT} – Source: {Stroke Centers, Emergency} – Method – Patient Position: supine – Method – Scan extent: C2-2-vertex – Method – Scan direction: caudocranial – Method – Respiration: suspended – Method – Acquisition-type: volumetric – Method – Contrast: {Non-contrast CT, PCT with contrast}
Redundancy	Multiple scans of the same patient are acceptable but need to be separated by one week.
Completeness	Each imaging scan should be accompanied by a radiology report that describes these features of the stroke: – Time from onset: { early hyperacute (0-6H), late hyperacute (6-24H), acute (1-7D), sub-acute (1-3W), chronic (3W+) } – CBV (Cerebral Blood Volume) in ml/100g of brain tissue – CBF (Cerebral Blood Flow) in ml/min/100g of brain tissue – Type of Stroke: {Hemorrhagic-Intracerebral, Hemorrhagic-subarachnoid, Ischemic-Embolic, Ischemic-Thrombotic}
Accuracy	Three reads of the image by separate radiologists to circumvent human errors and bias. Anonymized Patient history is sent to the radiologist.
Security and Privacy	Patient PII is not leaked to the radiologist interpreting the result or the researcher analyzing the data.

Data Quality Attributes

As you can see from the table of attributes for CT Stroke imaging data, the quality description is data-specific and use-specific.

Data engineers compute attribute-specific metrics using data attribute descriptions on a data sample to measure overall data quality. These attribute descriptions are the N* to pursue excellence in data quality.

Summary: The creation, collection, and correction improve over some time when measured using criteria. There will always be data quality blind spots and leakages. Hence, data engineers report data quality on a grayscale with multiple attribute-specific metrics.

Streaming vs. Messaging

“We already have pub/sub messaging infrastructure in our platform. Why are you asking for a streaming infrastructure? Use our pub/sub messaging infrastructure” – Platform Product Manager

Streaming and Messaging Systems are different. The use-cases are different.

Both streaming and messaging systems use the pub-sub pattern with producers posting messages and consumers subscribing. The subscribed consumers may choose to poll or get notified. Consumers in streaming systems generally poll the brokers, and the brokers push messages to consumers in messaging systems. Engineers use streaming systems to build data processing pipelines and messaging systems to develop reactive services. Both systems support delivery semantics (at least once, exactly once, at most once) of the messages. Brokers in streaming systems are dumber than messaging systems that build routing and filtering intelligence in the brokers. Streaming systems are faster than messaging systems due to a lack of routing and filtering intelligence 🙂

Let’s look at the top three critical differences in detail:

#1: Data Structures

In streaming, the data structure is a stream, and in messaging, the data structure is a queue.

“Queue” is FIFO (First In First Out) data structure. Once a consumer consumes an element, it is removed from the queue, reducing the queue size. A consumer cannot fetch the “third” element from the queue. Queues don’t support random access. E.g., A queue of people waiting to board a bus.

“Stream” is a data structure that is partitioned for distributed computing. If a consumer reads an element from a stream, the stream size does not reduce. The consumer can continue to read from the last read offset within a stream. Streams support random access; the consumer may choose to seek any reading offset. The brokers managing streams keep the state of each consumer’s reading offset (like a bookmark while reading a book) and allow consumers to read from the beginning, the last read offset, a specific offset, or the latest. E.g., a video stream of movies where each consumer resumes at a different offset.

In streaming systems, consumers refer to streams as Topics. Multiple consumers can simultaneously subscribe to topics. In messaging systems, the administrator configures the queues to send messages to one consumer or numerous consumers. The latter pattern is called a Topic used for notifications. A Topic in the streaming system is always a stream, and it’s always a queue in a messaging system.

Both stream and queue data structures order the elements in a sequence, and the elements are immutable. These elements may or may not be homogenous.

Queues can grow and shrink with publishers publishing and consumers consuming, respectively. Streams can grow with publishers publishing messages and do not shrink with consumers consuming. However, streams can be compacted by eliminating duplicates (on keys).

#2: Distributed (Cluster) Computing Topology

Since a single consumer consumes an element in a queue in a load-balancing pattern, the fetch must be from the central (master) node. The consumers may be in multiple nodes for distributed computing. The administrator configures the master broker node to store and forward data to other broker nodes for resiliency; however, it’s a single master active-passive distributed computing paradigm.

In the notification (topic) pattern, multiple consumers on a queue can consume filtered content to process data in parallel. The administrator configures the master node to store and forward data to other broker nodes that serve consumers. The publishers publish to a single master/leader node, but consumers can consume from multiple nodes. This pattern is the CQRS (Command Query Responsibility Segregation) pattern of distributing computing.

The streaming pattern is similar to the notification pattern w.r.t. distributed computing. Unlike messaging, partition keys break streams into shards/partitions, and the lead broker replicates these partitions to other brokers in the cluster. The leader election process selects a broker as a leader/master for a given shard/partition, and shard/partition replications serve multiple consumers in the CQRS pattern. The consumers read streams from the last offset, random offset, beginning, or latest.

If the leader fails, either a passive slave can take over, or the cluster elects a new leader from existing slaves.

#3: Routing and Content Filtering

In messaging systems, the brokers implement the concept of exchanges, where the broker can route the messages to different endpoints based on rules. The consumers can also filter content delivered to them at the broker level.

In streaming systems, the brokers do not implement routing or content filtering. Consumers may filter content, but utility libraries in the consumer filter out the content after the broker delivers the content to the consumer.

Tabular Differences View

Category	Streaming	Messaging
Support Publish and Subscribe Paradigm	Yes	Yes
Polling vs. Notification	Polling by Consumers	Notification by Brokers to consumers
Use Case	Data Processing Pipelines	Reactive (micro)services
Delivery Semantics Supported	at-most-once at-least-once exactly-once	at-most-once at-least-once exactly-once
Intelligent Broker	No	Yes
Data Structure	Stream	Queue
Patterns	CQRS	Content-Based Routing/Filtering Worker (LB) Distribution Notification CQRS
Data Immutability	Yes	Yes
Data Retention	Yes. Not deleted after delivery.	No. Deleted after delivery.
Data compaction	Yes. Key de-duplication.	N/A
Data Homogeneity	Heterogenous by Default. Supports schema checks on data outside the broker.	Heterogenous by Default.
Speed	Faster than Messaging	Slower than Streaming
Distributed Computing Topology	Broker cluster with single master per stream partition and consumers consuming from multiple brokers with data replicated across brokers	Broker cluster with single master per topic/queue. Active-passive broker configuration for the load-balancing pattern. Data replicated across brokers for multiple consumer distribution.
State/Memory	Brokers remember the consumers’ bookmark (state) in the stream	Consumers always consume from time-of-subscription (latest only)
Hub-and-Spoke Architecture	Yes	Yes
Vendors/Services (Examples)	Kafka Azure Event Hub AWS Kinesis	RabbitMQ Azure Event Grid AWS SQS/SNS
Domain Model	A stream of GPS positions of a moving car	A queue to buy train tickets

Table of Differences between Streaming/Messaging Systems

Visual Differences View

Summary

Use the right tool for the job. Use messaging systems for event-driven services and streaming systems for distributed data processing.