Python

Python Basics

Python Virtual Environments Upgrade Poetry in Python pprint() function Check Python versions on Mac Measure the execution time of code Linked lists Function statistics.fmean() Data Types Cheat Sheet Retry On Exception Defining Functions with Type Hints Generic Types Upgrade all packages in venv Use Code Runner with venv Generators Poetry for Python Package Management Renaming Python Virtual Env Practical Guide to Type Aliases in Python Python's Union Type Guide Understanding Python's 'Any' Distinct Types with NewType Python typing.Concatenate Python ClassVar Examples Using TypeGuard in Python Understanding 'Never' Type in Python Access CLI Arguments in Python Type Hints in Python Classes Using NoReturn in Python Typing Functions with Defaults Type Casting in Python Typing Multi-return Functions Typing with *args & **kwargs Annotating Nullable in Python Python Type Hints & Enum Variable Annotations in Python Type Hints in Exception Handling Unpacking Tuples with Types Python Typing: Examples Practical Guide: TypeVar in Python Recursive Types in Python Dependent Types in Python Install Python Modules Without Admin Override Methods in Python

Numbers & Math

Round a number Convert a number to hex value Convert a integer to binary Convert a float to binary Get the size of a number in bytes Check if a string can be converted to a number Extract numbers from text Get numerical value from user input Generate a random integer between min and max Generating a random float between min and max Reverse the order of digits in a number Format large numbers with comma separators Convert a number to a byte array Calculate Exponents

Strings

The modern Python strings cheat sheet The modern Python regular expressions cheat sheet Find the length of a string String slicing Ways to format a string Remove leading/trailing whitespaces Reverse a string Concatenate strings Check if a string contains a substring Convert a string to lowercase/uppercase Convert a string into a list Declare a multiline string Capitalize the first letter of each word in a string Check if a string is empty Count the number of words in a string Compare 2 strings ignoring case sensitivity Raw strings String replace() method Remove one or many substrings from a string Remove all non-alphanumeric characters from a string Convert a string to a URL slug Convert a string to bytes and vice versa Convert a string to integer char codes Remove consecutive spaces Convert a string to binary codes Get the size of a string in bytes Convert a character to a code point and vice versa Base64 encode/decode a string in Python Add leading zeros to a string Check if a string is a valid URL Extract all URLs from text Get the filename and the file extension from a URL Get hostname, domain, and protocol from a URL Align a string Reverse the order of wrods in a string Get a list of unique words from a string Replace unwanted words in a string with asterisks Convert a string to a hexadecimal value Ways to generate random strings Count the frequency of each word in a string Generate random color codes Validate phone numbers Remove HTML tags from a string Unescape HTML entities from a string Remove empty lines from a string Validate an email address Regex-Matched String Type Hint

Control flow

for loop While loop Conditional expression The WITH statement Try, except, else, finally The match/case statement Return multiple results from a function Keyword & positional arguments

Lists

Create a list List comprehension Access elements in a list Iterate through a list Count elements in a list Check whether a list is empty Replace/update elements in a list Combine lists Select random elements from a list Remove elements from a list Find all occurrences of a value in a list Reverse a list Find min and max in a numeric list Find the sum of a list Calculate the average of a numeric list List insert() method Sorting lists Create nested lists Flatten a nested list Clone a list Remove duplicates from a list Filter a list reduce() function Find common elements in 2 lists Swap 2 elements in a list Compare 2 lists Check if a list contains an element Shuffle a list Calculate the product of a numeric list Count the occurrences of elements in a list Iterate over 2 lists in parallel The map() function Divide a list into equally sized chunks Declaring lists with type hints Passing a list to a function as multiple arguments Generate a dummy list with N random elements List element-wise operations in Python IndexError: List index out of range Typing Mixed Lists in Python Generics in Multidimensional Lists Type Hints with Python map()

Dict, Set, Tuple

Python Dict vs JS Object Creating Dictionaries in Python Iterating Dictionaries in Python Accessing & Modifying Dictionaries Cloning Python Dictionaries Key Order in Python Dicts Adding to Python Dictionaries Removing Items from Dictionary Python Dictionary Views Explained Reverse Iterator over Dict Keys Checking Key Existence in Dictionary Checking Value in Dictionary Create JSON Data from a Dictionary Setting Default Dict Values in Python Variables as Dictionary Keys Merging Dictionaries in Python Compare 2 Dictionaries in Python Counting Items in Python Dictionaries Update Nested Dicts in Python Type Hints with Python Dicts Get Length of Python Dictionary Delete Dict Items Using Keys Sort Dictionary in Python Update List in Dict: Examples String to Dictionary in Python Convert Dictionary to String Max items in Python dict Rename Dict Key in Python Read CSV to Dictionary in Python Check if Dictionary is Empty Dict to Class Object in Python Random Dict Value Python Dataclass to Dict Fixing Python Dict Update Error Fixing 'dict' Object 'append' Error Fixing 'dict' has no 'iteritems' Random Item from Dictionary Pretty Print Nested Dictionaries Fixing Python RuntimeError Python: Get First/Last Dict Item Passing Dictionaries in Python Diff 2 Dicts in Python Set vs FrozenSet in Python Creating Sets in Python Checking dict subsets in Python Comparing Sets in Python Set Union & Intersection in Python set symmetric_difference_update() Set difference_update() Method Converting Sets and Lists in Python Maintaining Order in Python Sets Adding Elements to Python Sets Getting Elements from Python Sets Intersect N Sets in Python Creating Tuples in Python Python Range Type Guide Checking Range Overlaps in Python Named Tuples in Python Comparing 2 Tuples in Python Sorting Tuples in Python List to Tuple Conversion in Python Type Hints with Tuples in Python Merging Tuples in Python Combine N Lists to Tuples Unzipping Lists of Tuples Filtering List of Tuples in Python Count Occurrences in Tuple List of Tuples to Dict Python List of Tuples Typing Convert Tuple <-> String in Python Looping Through Tuples in Python Using Aliases for Tuple Returns Unpack Tuples Pythonically Creating Empty Tuples in Python Convert a dictionary to a query string Generic Types for Nested Dicts Type Hints & Namedtuple in Python

Network & JSON

Parsing JSON data List to JSON file Convert a list to JSON Fetching GitHub Data Offline Py Packages Set Headers in Python Request params Setting Cookies POST JSON with Python File Upload in Python Python File Download Python File Download Fix Request SSLError Send GET requests with aiohttp Python API Requests aiohttp POST requests Python Requests Proxy Set timeouts when using aiohttp Async Python Requests Download files with aiohttp streams Requests Timeout aiohttp upload files Custom User Agent Fix ImportError aiohttp - send multiple requests concurrently Handling aiohttp Errors Parse HTML with Python Create a simple web server with aiohttp Python Crawling 101 Fix aiohttp Error Custom Headers aiohttp aiohttp Parameters Rate Limit aiohttp Limit aiohttp Requests Fix aiohttp Error Aiohttp Mimetype Fix Fix aiohttp Network Error aiohttp Logging Fix aiohttp Warning aiohttp API Auth Requests Exceptions Python Requests Python Requests 101 Use aiohttp to crawl webpages asynchronously Python HTTPError Fix Fix Timeout Errors Fix TooManyRedirects Fix ConnectionError Fix RequestException Python JSON Requests Auto retry in Requests Fixing Python 403 Error Disable Req Warnings Fix MaxRetryError Silence 'requests' Logs Fix Python SSL Error Fixing SSL Error Fix No Conn. Adapter Fix SSLError in Python Fix SSL Error in Python FTP File Download requests vs aiohttp Python CSRF Tokens Python XML Requests Handling CSV in Python Block Python Requests Force IPv4/IPv6 in Requests Python File Upload Python httpx Requests Set httpx Timeout Python httpx Headers Best open-source libraries to make HTTP requests

System & File I/O

File modes OS Info in Python Zip/Unzip in Python Resize Images in Py File Creation Date Python Folder Size Iterate Directory Async File I/O in Python Async CSV in Python

Date and Time

Convert Datetime to Timestamp and vice versa Comparing 2 Dates Convert Date Strings into Date Objects Modify datetime objects in Python Calculate Age from ISO 8601 Date Format Date and Time Check if a date string is valid Get the Current Date and Time with Timezone Get a list of dates between 2 dates Modify Timestamps in Python Datetime to Hex in Python Convert Timestamp to Bytes in Python DST in Python: A Guide Date String to Timestamp Convert Time Formats in Python Extract Date from Datetime Check Date Range Overlap Checking Leap Year in Python Group Dates by Year & Month in Python Check Date Between Two Dates Get Day from Timestamp in Python Convert Seconds in Python UTC to Local Time in Python Event Countdown in Python Sort User Dicts by DOB in Python Loop Through Dates in Python Datetime ISO & RFC Conversion Datetime to Base64 in Python Truncate Milliseconds in Python Find Median Date in Python Generate datetime from calendar Python: Month/Day from Number Display Full Year Calendars Custom Monthly Calendars in Python Find Weekday for Any Date in Python Generate Weekly Calendars in Python Listing Leap Years in Python Creating HTML Calendars in Python Interactive CLI Calendar in Python Calculate US public holidays Python Calendars to CSV Type Hints with DateTime

Async & Await

An introduction to coroutines and event loops asyncio.run() Async/await and loops asyncio.sleep() function asycnio.create_task() function asyncio.wait_for() function asyncio.gather() function async/await & timeouts loop.run_until_complete() Define and call async functions Handle exceptions Use the result returned by an async function asyncio.Runner() context manager asyncio.Semaphore class Running a function periodically with asyncio asyncio.wait() function asyncio.as_completed() function asyncio.Queue class Async generators (with "yield") run_coroutine_threadsafe() Using async/await with class methods Define a class with an async constructor asyncio.loop.run_in_executor() Run a task at a certain time time every day loop.create_server() method Add a coroutine to an already running event loop Thread-Safety in Python asyncio.open_connection() Explained Python AsyncIO Socket Server Creating Future Objects in Python asyncio.StreamReader Practical Guide Asyncio Callbacks in Python Python asyncio.StreamWriter Guide Returning Values from Futures Dataclass with SQLite3 in Python Convert Future to Task in Python 3.11+ Converting Callbacks to Async in Python asyncio.open_connection() Explained AsyncIO on Multi-core CPUs with Python AsyncIO for Sequential Downloads Asyncio Parallel File Downloads Asyncio File Download Progress Asyncio TCP Server in Python Type Hinting Async Functions When to Avoid asyncio in Python Async Generator Functions in Python Asyncio.Lock in Python - Explained Python and asyncio.run() issues Exploring asyncio.Barrier Fix aiohttp Wheel Error Exploring asyncio.Event in Python Python asyncio & Shell Commands Asyncio Event Loop Fix Python asyncio & Subprocess Fix Asyncio RuntimeError AsyncIO Subprocess Exec in Python Popular async Python libraries Python asyncio & subprocesses Asyncio: Stop/Kill Child Process Listing Undone asyncio Tasks Preventing asyncio Task Cancellation asyncio.timeout_at() in Python Naming asyncio Tasks in Python Asyncio: Run Functions in Threads Determine Running asyncio Task Check asyncio Task Cancellation Identify asyncio Event Loop Thread Managing Tasks with asyncio.TaskGroup Asyncio Priority Queue in Python Asyncio Loading Indicator in Python Fix asyncio.run() Error in Python Limiting Concurrent asyncio Tasks Async Functions & 'with' in Python Fix asyncio dict await error Clean up resources in Python Iterating with 'async for' in Python Async Resource Management in Python Python asyncio & JS Promise Chaining Python asyncio Task Wrapping RuntimeWarning: Coroutine was never awaited CancelledError exception SyntaxError: 'await' outside async function

Dataclasses

Introduction to dataclass Defining Fields in Dataclass Using Dataclass Methods Using Inheritance with Dataclass How to Validate Data in Dataclass Set Default Values in Dataclass Using Class Decorators with Dataclass

Data Persistence

Python sqlite3: Connect & Close Python sqlite3: Connection Timeout Python sqlite3 Cached Statements Fixing sqlite3 ProgrammingError Python sqlite3 Type Conversion Fix sqlite3 Single Statement Warning Fixing sqlite3 Numeric Errors Fixing sqlite3 Size Limit Error Fix sqlite3 Database Path Error Fixing sqlite3 Locked Database Fixing sqlite3.OperationError Fixing sqlite3 IntegrityError sqlite3.NotSupportedError Fixes SQLite Transactions in Python Python sqlite3 & Placeholders Python sqlite3: Insert & Get ID Python sqlite3 Bulk Inserts Python sqlite3 Auto-Increment PK Python sqlite3 Upsert Tutorial Sqlite3 Connection Runtime Limit Python sqlite3 & Execution Methods Python sqlite3 & create_function() Python sqlite3 create_aggregate sqlite3 create_window_function Guide sqlite3 create_collation() in Python 3.11 sqlite3 iterdump() & backup() SQLite3 Serialization in Python Python sqlite3 Fetch Methods Python sqlite3: Get Column Names sqlite3.Blob in Python 3.11 Python sqlite3 Adapters Python sqlite3 in Read-Only Mode SQLite3 Pagination in Python Handling UTF-8 in sqlite3 with Python Fix _sqlite3 ModuleError in Python SQLite to JSON in Python SQLite to CSV in Python Python sqlite3 CRUD Removing Duplicates in sqlite3 SQLite3 Aggregation in Python Python sqlite3: Select & Count Rows Choosing SQLite Date Time Format Python & Multiple SQLite DBs PyMongo Connection Guide PyMongo Connection Pooling List Databases with PyMongo Listing Collections with PyMongo PyMongo: Insert Document & Get ID PyMongo Bulk Insert PyMongo Latest/Earliest Doc PyMongo: find_one() and find() Count Documents with PyMongo Sorting Documents in PyMongo Sorting with PyMongo PyMongo Pagination Guide PyMongo: Select Documents in Range Indexing in PyMongo Guide Executing Raw DB Commands in PyMongo sqlite3 & Type Hints in Python PyMongo with Type Hints sqlite3 Query Timing in Python PyMongo: Update & Delete Docs Translate MongoDB to PyMongo PyMongo Aggregation Pipeline PyMongo: Distinct Docs PyMongo: Array Filters Update PyMongo: Nested Fields Ops Custom Types in PyMongo Save datetime with timezone PyMongo: Grouping Documents PyMongo: Location Data Handling Insert & Update JSON with PyMongo Fixing PyMongo Connection Error Querying with Regex in PyMongo Simulate LIKE/ILIKE in PyMongo Search Docs by ObjectId in PyMongo Find Docs by List Values in PyMongo Fixing PyMongo ValueError ObjectId Conversion in PyMongo Fixing PyMongo CursorNotFound Error Closing PyMongo Connections Fixing PyMongo Duplicate Key Error Fixing PyMongo Upsert Error PyMongo Upsert Examples Fixing PyMongo Collection Error Find Nulls with PyMongo PyMongo to Dict List PyMongo: Create/Drop Indexes PyMongo: Create/Drop DBs PyMongo Date-based Queries PyMongo Geospatial Queries PyMongo $ Operator Error Fix Cursor Length in PyMongo Connect to MongoDB with PyMongo PyMongo Auth Examples PyMongo: ISODate Operations Excluding _id in PyMongo Queries Fix ImportError: 'pymongo' Check Collection Existence PyMongo: Removing Fields Appending Values with PyMongo Fixing PyMongo DateTime Error PyMongo: Set Connection Timeout Fixing PyMongo BSON Import Error Case-Insensitive Search with PyMongo PyMongo ObjectId generation_time Fix: PyMongo Exceeded Memory Error Creating Compound Indexes Fix PyMongo InvalidDocument Error PyMongo: $and, $or, $not, $nor Ops Check & Upgrade PyMongo Version Solving BulkWriteError in MongoDB Connect to MongoDB via MongoEngine Multi-DBs with MongoEngine MongoEngine: Document Schema MongoEngine ListField Guide MongoEngine & EnumField MongoEngine: Image & FileField MongoEngine Indexing Guide MongoEngine Doc Validation Guide Sorting Documents with MongoEngine MongoEngine CRUD Tutorial MongoEngine Text Search MongoEngine Pagination Basics Ignore Fields in MongoEngine Exclude Fields in MongoEngine Insert Documents with MongoEngine Update Document by ID in MongoEngine Delete Docs By Condition in MongoEngine MongoEngine Upsert Tutorial MongoEngine: AND & OR Queries MongoEngine: Filter Documents Updating Embedded Docs in MongoEngine Convert MongoEngine Doc to JSON Regex in MongoEngine Fixing MongoEngine ConnectionError Fetch Latest/Earliest with MongoEngine MongoEngine: Create/Drop Collection Close MongoEngine Connection MongoEngine Aggregation Guide Query Distinct Values with MongoEngine MongoEngine Default Values MongoEngine Field Essentials MongoEngine: Set Text Field Max Length Set min/max in MongoEngine MongoEngine: Unique Fields MongoEngine: Query Insights MongoEngine ReferenceField Guide Appending to ListField in MongoEngine MongoEngine DateTimeField Guide MongoEngine BinaryField Guide MongoEngine DictField Guide

Fun Examples

Get all links from a webpage with Beautiful Soup Extract plain text from a webpage Extract and download all images from a webpage 3 ways to hash a password Auto-Tweet with Py Reddit Bot Tutorial Python Twitter Bot Python & Selenium Python Telegram Bot Check System RAM/CPU/Disk Usage Schedule events with sched.scheduler class Crawl HTML Tables & Save to CSV Pillow Image Generation Pillow Watermarks Text on Image w/ Pillow Python & Faker Faker for Python Python & Faker: Text Length Control Faker for i18n text in Python Py Unsplash API - Photo Download Extract Tables from HTML Retrieve Location from IP in Python Run Git Commands with Python Using Optional Type in Python

Common errors

Buffer Interface Error Fix int TypeError Fixing IndentationError Fix AttributeError TypeError Fix Fixing 'NoneType' AttributeError Fixing Python RecursionError Fixing Circular Ref Fix Coroutine Error Fix ValueError UnboundLocalError Fix Fixing TypeError Fixing KeyError Fixing Python Secure Coding Warning Fix Python NameError: List Not Defined Fixing 'NoneType' has no len() Error Fixing TypeError: str not bytes

Python: Get Hostname and Protocol from a URL

Updated: June 2, 2023 By: Khue Post a comment

This concise article shows you how to get the hostname and the protocol from a given URL in Python.

Before writing code, let me clarify some things. Suppose we have a URL like this:

https://www.slingacademy.com/cat/sample-data

Then the components of the URL are:

https: The protocol.
www.slingacademy.com: The hostname.
www: The subdomain.

slingacademy.com: The domain (domain name).
/cat/sample-data: The path segment.

Table Of Contents

1 Using the urllib.parse module

2 Using regular expressions

Using the urllib.parse module

The built-in urllib.parse module of Python provides beautiful tools for URL parsing. It can handle various URL formats, including different protocols.

Here’re the steps to extract the hostname and protocol from a URL:

Import the urlparse function from the urllib.parse module.
Parse the URL using the urlparse function.

Access the hostname and scheme attributes from the parsed URL object.

A code example is worth more than thousands of boring words:

from urllib.parse import urlparse

def extract_hostname_and_protocol(url):
    parsed_url = urlparse(url)
    hostname = parsed_url.hostname
    protocol = parsed_url.scheme
    return hostname, protocol


# Test it out
url1 = 'https://www.slingacademy.com/cat/sample-data/'
hostname_1, protocol_1 = extract_hostname_and_protocol(url1)
print(f"Hostname: {hostname_1}")
print(f"Protocol: {protocol_1}")

url2 = "https://api.slingacademy.com/v1/examples/sample-page.html"
hostname_2, protocol_2 = extract_hostname_and_protocol(url2)
print(f"Hostname: {hostname_2}")
print(f"Protocol: {protocol_2}")

Output:

Hostname: www.slingacademy.com
Protocol: https
Hostname: api.slingacademy.com
Protocol: https

Using regular expressions

The preceding approach is elegant and works well. However, it isn’t the only possible way to get the job done. An alternative solution is to use a regular expression.

The steps are:

Import the re module for regular expressions.

Define a regular expression pattern to match the hostname and protocol.
Use the re.search function to find the pattern within the URL string.
Extract the matched groups for hostname and protocol.

Here’s the pattern we’ll use:

pattern = r"^(?P<protocol>https?)://(?P<hostname>[^/]+)"

Let me explain the pattern above:

^ : Start of the string anchor.

(?P<protocol>https?) : Named capturing group protocol to match the protocol. It matches http or https using the ? quantifier to make the s optional.
://: Matches the colon and double slashes.
(?P<hostname>[^/]+) : Named capturing group hostname to match the hostname. It matches one or more characters that are not a forward slash (/), indicating the hostname portion of the URL.

Code example:

import re

def extract_hostname_and_protocol(url):
    pattern = r"^(?P<protocol>https?)://(?P<hostname>[^/]+)"
    match = re.search(pattern, url)
    if match:
        protocol = match.group("protocol")
        hostname = match.group("hostname")
        return hostname, protocol
    return None, None


# Test it out
url1 = 'https://www.slingacademy.com/cat/sample-data/'
hostname_1, protocol_1 = extract_hostname_and_protocol(url1)
print(f"Hostname: {hostname_1}")
print(f"Protocol: {protocol_1}")

url2 = "http://api.slingacademy.com/v1/examples/sample-page.html"
hostname_2, protocol_2 = extract_hostname_and_protocol(url2)
print(f"Hostname: {hostname_2}")
print(f"Protocol: {protocol_2}")

Output:

Hostname: www.slingacademy.com
Protocol: https
Hostname: api.slingacademy.com
Protocol: http

Regular expressions allow you to deal with rare and specific use cases by crafting your own custom pattern. However, it may be tough sometimes, even with experienced programmers.

Next Article: Python: 5 Ways to Reverse a String

Previous Article: Python: How to define a regex-matched string type hint

Series: Working with Strings in Python

Python