Posts

dbt (Data Build Tool) is fast gaining traction in the modern data platform architecture. Its primary function? To enable analysts and engineers to more efficiently transform data within their data warehouse. The DBT universe comprises two main variants: DBT Core: This open-source framework is dedicated to data transformation and is available at no cost. DBT Cloud: A managed service that offers Git-integrated code editing, job orchestration, and data quality controls, all built on the robust engine of DBT Core....

29/8/2023 Their is a common misconception that AWS Egress cost to internet is expensive( and it is) but did you know cross-region is not cheap either I have based this calculation based on AWS Sydney region as base region (Aug 29th 2023) via the cost calculator - for data transfer charges https://calculator.aws/#/estimate Region Transfer to Data Amount Cost Estimate per Month AWS Sydney region Internet First 100 GB FREE AWS Sydney region Internet 1 TB 132 USD AWS Sydney region AWS Melbourne Region 1 TB 102 USD AWS Sydney region Other Region 1 TB 120....

Interact with your Gmail with Agent GPT 25/8/2023 Idea: Sometimes we signup for too many news letters and information digests across different form and give our email. Idea is to create summary from newsletters i received based on day/author and create a summary email send to you How i POCed it leverage Gmail API and Langchain agent https://python.langchain.com/docs/integrations/toolkits/gmail Learning Getting gmail tkoen from credentials.json require some troubleshooting Summarise multiple emails usecase is not that straight forward You may try like read a particular email, create draft etc as interim usecases....

Apache Iceberg — Solution Considerations Apache Iceberg is been buzzing for a while this post is to deep dive on what that means and its solution considerations What is it? Its a table format to store and interact with the Data From user perspective who are familiar in SQLs it is a way to answer the question of what data is in this table? , its not the data that’s going to be in some database, but stored in a file that reside in a data lake (like s3) and then users can interact with that table with SQL....

Evaluation and Testing LLM Outputs This is one of series of articles on different solution considerations for integrating into LLMs. Question How can we evaluate and test the Large Language model outputs This is critical question among enterprises for integrating LLMs into production Their is no single answer here. Integrating to LLMs has been widespread already , their are lots of showcases, demo tutorials etc. However not many stories on integrating output response from LLMs directly into actions or to end users, especially when your business is at stake....