52 lines
1.9 KiB
Diff
52 lines
1.9 KiB
Diff
From 6e0231074111ff1fda0b2de1765e1988fda2e9c9 Mon Sep 17 00:00:00 2001
|
|
From: Brendan Cunningham <bcunningham@cornelisnetworks.com>
|
|
Date: Fri, 7 May 2021 16:33:35 -0400
|
|
Subject: [PATCH] README: add section about OMPI 4.1.x, OFI BTL, and high-PPN
|
|
jobs.
|
|
|
|
Signed-off-by: Brendan Cunningham <bcunningham@cornelisnetworks.com>
|
|
---
|
|
README | 19 ++++++++++++++++++-
|
|
1 file changed, 18 insertions(+), 1 deletion(-)
|
|
|
|
diff --git a/README b/README
|
|
index 7990555..2961534 100644
|
|
--- a/README
|
|
+++ b/README
|
|
@@ -67,7 +67,6 @@ Contains the following sections:
|
|
- INSTALLING
|
|
* INSTALLING USING MAKEFILE
|
|
* INSTALLING USING EITHER YUM OR DNF
|
|
-- TESTING
|
|
- RELATED SOFTWARE TO PSM2
|
|
- SUPPORTING DOCUMENTATION
|
|
|
|
@@ -251,6 +250,24 @@ libraries available on them. Open MPI provides a standard configure, make and
|
|
make install mechanism which will detect and build the relevant PSM2 network
|
|
modules for Open MPI once the header and runtime files are detected.
|
|
|
|
+Open MPI 4.1.x, OFI BTL, and high PPN jobs
|
|
+----------------
|
|
+Open MPI added the OFI BTL for one-sided communication. On an OPA fabric, the
|
|
+OFI BTL may use the PSM2 OFI provider underneath. If PSM2 is in-use as both
|
|
+the MTL (directly or via OFI) and the BTL (via OFI), then each rank in the
|
|
+Open MPI job will require two PSM2 endpoints and PSM2 context-sharing will
|
|
+be disabled.
|
|
+
|
|
+In this case, total number of PSM2 ranks on a node can be no more than:
|
|
+ (num_hfi * num_user_contexts)/2
|
|
+Where num_user_contexts is typically equal to the number of physical CPU
|
|
+cores on that node.
|
|
+
|
|
+If your job does not require an inter-node BTL (e.g. OFI), then you can
|
|
+disable the OFI BTL in one of two ways:
|
|
+ 1. When building Open MPI, specify '--with-ofi=no' when you run 'configure'.
|
|
+ 2. When running your Open MPI job, add '-mca btl self,vader'.
|
|
+
|
|
MVAPICH2 support
|
|
----------------
|
|
MVAPICH2 supports PSM2 transport for optimized communication on HFI hardware.
|
|
--
|
|
2.32.0
|
|
|