3 Reasons Data Engineers Are the Unsung Heroes of GenAI
Ultimately, the success of GenAI depends on data quality. Without accurate,
reliable data consistently made available to LLMs, even the most advanced models
won’t produce useful outputs. Over the last five years, leading data engineers
have adopted observability tooling — including automated monitoring and
alerting, similar to DevOps observability software — to help improve data
quality. Observability helps data teams monitor and proactively respond to
incidents like failed Airflow jobs, broken APIs and misformatted third-party
data that put data health at risk. And with end-to-end data lineage, teams gain
visibility into upstream and downstream dependencies. Data engineers can provide
transparency when observability tooling is applied across the modern AI stack,
including vector databases. Lineage allows engineers to trace the source of the
data as it’s converted to embeddings, then use that data to generate rich text
that the LLM puts in front of the user.
Why you’ll soon have a digital clone of your own
Far more interesting than an avatar that looks and acts like you is one that
thinks and communicates like you — a virtual you with an AI brain for
interacting with others on your behalf. Meta is working on tech called
“Creator A.I.” that will enable real Instagram influencers to create fake
digital AI versions of themselves to interact with fans through direct
messages and comments. That initiative is a glimpse of the near future of
business communication. We’ve been talking about “digital transformation” for
a decade. But it’s only recently when that transformation involved digitizing
ourselves. Businesses are now looking to embrace the concept of digital
avatars for all the same reasons as other digital transformation initiatives:
Higher productivity and lower costs. This process involves the cloning of
existing people. Any day now, an industry will emerge where your face and body
are scanned, your voice is recorded and your communications are fed into the
system, so it knows how you use words. From that point, a virtual version of
you can leave high-resolution video messages from a simple command you give to
your AI glasses.
Euroviews: 'Quiet hiring' is silently revolutionising the workplace
Quiet hiring is gaining traction as a strategic alternative to traditional
recruitment methods. Instead of constantly expanding the workforce through
external hires, companies are increasingly focusing on upskilling and
retraining the talent right under their noses, to meet evolving business
needs. ... Quiet hiring also encourages a culture of continuous learning and
development within the workforce, driving employees to proactively seek out
and pursue professional growth opportunities. This is done by empowering
existing talent, encouraging them to explore different roles and projects
within the company, and investing in their development. By taking these steps,
management teams can mitigate talent shortages, all while also driving
employee loyalty, satisfaction, and retention. Tangible skills have never been
more important. In fact, our latest research reveals that companies are
integrating intentional learning with key skills like leadership,
collaboration, and analytical thinking, aiming to both nurture professional
growth and cultivate a supportive team culture.
Understanding emerging AI and data privacy regulations
The AI Act is the first comprehensive AI regulation adopted by EU lawmakers.
It imposes obligations on both providers and deployers of AI systems, and
specifically targets practices and systems that pose some risk to the health
and safety or the fundamental rights of individuals, with the purpose of
ensuring “a high level of protection of health, safety, fundamental rights
(…), including democracy, the rule of law and environmental protection,
against the harmful effects of AI systems.” ... The second category
encompasses high-risk systems for which providers are subject to a whole set
of requirements including risk management, data governance, record-keeping,
transparency, human oversight, accuracy, robustness, and cybersecurity.
... The third category encompasses certain AI systems that are subject to
transparency requirements to make sure the public is in a position to
recognize artificially generated or manipulated outputs. The fourth category
encompasses other systems which are largely unregulated. Importantly,
general-purpose AI models and systems are also in scope.
Indian SaaS Market: Good times ahead
There is a substantial demand for SaaS solutions from Small and Medium
Businesses (SMBs) seeking to embrace digitisation and enhance their market
reach and customer experience. SaaS companies must adopt a mobile-first
strategy and tailor their go-to-market strategies to address the largely
untapped SMB segment to increase adoption and ease of usage. In India, where
personal connections play a significant role, human interaction will be
crucial in selling the product. While horizontal SaaS solutions (e.g., CRM,
HR, etc.) have traditionally dominated the market, there is a growing
preference for vertical-specific solutions tailored to specific industries
such as healthcare, education, finance, and logistics. Vertical SaaS companies
stand to gain a loyal customer base by offering integrated solutions aligned
with users’ systems and workflows. This trend is likely to gain further
traction in 2024 as companies look for more specialised solutions. The
emergence of Generative AI presents new opportunities and challenges for SaaS
solutions providers. With a large pool of AI talent in India, SaaS companies
can capitalise on this trend to develop innovative categories of services and
revenue streams.
Maryland Legislature Passes State Privacy Bill With Robust Requirements And Broad Threshold For Application
MODPA prohibits the sale of sensitive data, which is a subset of personal data.
A “sale of personal data” means the “exchange of personal data by a controller,
a processor, or an affiliate of a controller or processor to a third party for
monetary or other valuable consideration.” “Sensitive data” includes data
revealing: racial or ethnic origin; religious beliefs; consumer health data; sex
life; sexual orientation; status as transgender or nonbinary; national origin;
and citizenship or immigration status. It also includes genetic and biometric
data, personal data of a consumer who the controller knows or has reason to know
is a child, and precise geolocation data. ... MODPA imposes guardrails with
respect to the processing and sale of minors’ personal data. Controllers are
prohibited from selling personal data of a consumer or using that data for
purposes of targeted advertising if the controller knew or should have known
that the consumer is under the age of 18. This prohibition is strict compared to
other laws that require actual knowledge of consumers’ age or provide an
opportunity for consumers to opt-in for the processing and sale of minors’ data.
Expectations vs. reality: A real-world check on generative AI
There are gen AI success stories in verticals like document engineering, where
Docugami offers custom small language models that build a knowledge graph from a
customer’s own complex documents, and can be used for both document generation
and to extract data. And commercial insurance is a vertical Docugami CEO Jean
Paoli says has been an early adopter, including statements of value,
certificates of insurance, as well as policy documents with renewal dates,
penalties, and liabilities. That’s critical information describing the risk of
both individual customers and the entire portfolio, which has been difficult to
manually extract and consolidate to use for generating new quotes, or
representing the portfolio to reinsurers. “These are real scenarios that save
you millions of dollars, not few hundred bucks,” Paoli says. Like everyone else,
large Docugami customers created gen AI committees and started pilots in 2023,
but many have already moved from discovery to implementation, starting
production deployments at least six months ago and seeing real returns, chief
business officer Alan Yates says.
9 Steps Towards an Agile Architecture
The first step in breaking these limiting beliefs is to gain consensus that the
system will be released in a series of increments. The essence of the Minimum
Viable Product (MVP) approach is that each product increment will attempt to
deliver at least one outcome to at least a subset of the users of the system.
The organization will measure its success in achieving these outcomes and use
that feedback to decide what it should work on next. Using an MVP/MVA approach
changes the conversations that a development team and its stakeholders have. ...
Learning to develop architecture incrementally takes practice, and no team is
going to master it immediately. Most teams lean toward developing more
architecture than they need, only to find that some of what they did was wrong,
or at least unnecessary. They will probably struggle to balance trade-offs. They
will also probably learn that some of the choices they made are wrong. All of
this is good. They have to learn the right balance on their own, and they can
only do this by making some wrong choices and seeing the impact.
Tracking Data Breaches: Targeting of Vulnerabilities Surges
To penetrate corporate networks, hackers most often gained unauthorized access
to a victim's web applications, including cloud-based email or collaboration
tools, Verizon said. The next most common ways that attackers broke in involved
phishing emails, followed by exploiting vulnerabilities in web applications,
stealing credentials for desktop sharing software, and using stolen VPN
credentials. For the first time, Verizon's annual report counts software
vulnerabilities - such as the widely exploited MOVEit bug - as a supply chain
security metric. "We want to give organizations the ability to see what they can
effect in their environment by choosing vendors who follow 'secure by design'
practices," Widup said. The Clop campaign targeting MOVEit was notable in part
because it seemed to run for only two or three days. While vendor Progress
Software quickly issued a patch, the latest count from security firm Emsisoft is
that Clop's blitzkrieg tactics directly or indirectly affected 2,770
organizations and exposed data pertaining to 95 million individuals.
How to Manage a Rapidly Growing IT Team
The key to keeping pace with rapid growth lies in structuring an IT team that
embraces new technology and training to stay current, says Diane Rafferty,
managing director, national technology group, at employment placement firm
Atrium, via email. "If you don't offer training on the most up-to-date
technologies, your team will have to do more with less." Leadership must clearly
define expectations, yet leave room for movement, Velco says. "Autonomy inspires
ownership of roles and collaborative problem-solving beyond defined bounds," he
explains. Structures alone don't scale, Velco notes, but processes do.
"Implementing agile frameworks like Scrum or Kanban encourages frequent
reassessment and adjustment." When it comes to preparing for team growth,
Daniele Gemignani, CTO at no-code business process automation solution provider
Pipefy, is a strong believer in mentorship and leadership development programs.
"For teams with a mix of seniority levels, pairing junior members with more
experienced mentors can accelerate their development and readiness to take on
more complex tasks or leadership roles," she says
Quote for the day:
"Don’t be distracted by criticism.
Remember — the only taste of success some people get is to take a bite out
of you." -- Zig Ziglar
No comments:
Post a Comment