Subject: [PATCH 2550/2944] refcount_t: Add ACQUIRE ordering on success for dec(sub)_and_test() variants

From: Elena Reshetova <[email-masked]>
Date: Wed, 30 Jan 2019 13:18:51 +0200

to #32314506

commit 47b8f3ab9c49daa824af848f9e02889662d8638f upstream.

This adds an smp_acquire__after_ctrl_dep() barrier on successful
decrease of refcounter value from 1 to 0 for refcount_dec(sub)_and_test
variants and therefore gives stronger memory ordering guarantees than
prior versions of these functions.

Co-developed-by: Peter Zijlstra (Intel) <[email-masked]>
Signed-off-by: Elena Reshetova <[email-masked]>
Signed-off-by: Peter Zijlstra (Intel) <[email-masked]>
Reviewed-by: Andrea Parri <[email-masked]>
Cc: Andrew Morton <[email-masked]>
Cc: Linus Torvalds <[email-masked]>
Cc: Paul E. McKenney <[email-masked]>
Cc: Peter Zijlstra <[email-masked]>
Cc: Thomas Gleixner <[email-masked]>
Cc: Will Deacon <[email-masked]>
Cc: [email-masked]
Cc: [email-masked]
Cc: [email-masked]
Link: https://lkml.kernel.org/r/[email-masked]
Signed-off-by: Ingo Molnar <[email-masked]>
Signed-off-by: Xuan Zhuo <[email-masked]>
Acked-by: Dust Li <[email-masked]>
---
 Documentation/core-api/refcount-vs-atomic.rst | 24 +++++++++++++++++++---
 arch/x86/include/asm/refcount.h               | 29 +++++++++++++++++++++++++--
 lib/refcount.c                                | 18 ++++++++++++-----
 3 files changed, 61 insertions(+), 10 deletions(-)

diff --git a/Documentation/core-api/refcount-vs-atomic.rst b/Documentation/core-api/refcount-vs-atomic.rst
index 322851b..976e85a 100644
--- a/Documentation/core-api/refcount-vs-atomic.rst
+++ b/Documentation/core-api/refcount-vs-atomic.rst
@@ -54,6 +54,13 @@ must propagate to all other CPUs before the release operation
 (A-cumulative property). This is implemented using
 :c:func:`smp_store_release`.
 
+An ACQUIRE memory ordering guarantees that all post loads and
+stores (all po-later instructions) on the same CPU are
+completed after the acquire operation. It also guarantees that all
+po-later stores on the same CPU must propagate to all other CPUs
+after the acquire operation executes. This is implemented using
+:c:func:`smp_acquire__after_ctrl_dep`.
+
 A control dependency (on success) for refcounters guarantees that
 if a reference for an object was successfully obtained (reference
 counter increment or addition happened, function returned true),
@@ -119,13 +126,24 @@ Memory ordering guarantees changes:
    result of obtaining pointer to the object!

-case 5) - decrement-based RMW ops that return a value
------------------------------------------------------
+case 5) - generic dec/sub decrement-based RMW ops that return a value
+---------------------------------------------------------------------
 
 Function changes:
 
  * :c:func:`atomic_dec_and_test` --> :c:func:`refcount_dec_and_test`
  * :c:func:`atomic_sub_and_test` --> :c:func:`refcount_sub_and_test`
+
+Memory ordering guarantees changes:
+
+ * fully ordered --> RELEASE ordering + ACQUIRE ordering on success
+
+
+case 6) other decrement-based RMW ops that return a value
+---------------------------------------------------------
+
+Function changes:
+
  * no atomic counterpart --> :c:func:`refcount_dec_if_one`
  * ``atomic_add_unless(&var, -1, 1)`` --> ``refcount_dec_not_one(&var)``
 
@@ -136,7 +154,7 @@ Memory ordering guarantees changes:
 .. note:: :c:func:`atomic_add_unless` only provides full order on success.

-case 6) - lock-based RMW
+case 7) - lock-based RMW
 ------------------------
 
 Function changes:
diff --git a/arch/x86/include/asm/refcount.h b/arch/x86/include/asm/refcount.h
index 19b9052..c0efb6b 100644
--- a/arch/x86/include/asm/refcount.h
+++ b/arch/x86/include/asm/refcount.h
@@ -65,18 +65,43 @@ static __always_inline void refcount_dec(refcount_t *r)
 }
 
 static __always_inline __must_check
-bool refcount_sub_and_test(unsigned int i, refcount_t *r)
+bool _refcount_sub_and_test(unsigned int i, refcount_t *r)
 {
 	GEN_BINARY_SUFFIXED_RMWcc(LOCK_PREFIX "subl", REFCOUNT_CHECK_LT_ZERO,
 				  r->refs.counter, "er", i, "%0", e, "cx");
 }
 
-static __always_inline __must_check bool refcount_dec_and_test(refcount_t *r)
+static __always_inline __must_check
+bool refcount_sub_and_test(unsigned int i, refcount_t *r)
+{
+	bool ret = _refcount_sub_and_test(i, r);
+
+	if (ret) {
+		smp_acquire__after_ctrl_dep();
+		return true;
+	}
+
+	return false;
+}
+
+static __always_inline __must_check bool _refcount_dec_and_test(refcount_t *r)
 {
 	GEN_UNARY_SUFFIXED_RMWcc(LOCK_PREFIX "decl", REFCOUNT_CHECK_LT_ZERO,
 				 r->refs.counter, "%0", e, "cx");
 }
 
+static __always_inline __must_check bool refcount_dec_and_test(refcount_t *r)
+{
+	bool ret = _refcount_dec_and_test(r);
+
+	if (ret) {
+		smp_acquire__after_ctrl_dep();
+		return true;
+	}
+
+	return false;
+}
+
 static __always_inline __must_check
 bool refcount_add_not_zero(unsigned int i, refcount_t *r)
 {
diff --git a/lib/refcount.c b/lib/refcount.c
index ebcf8cd..6e904af 100644
--- a/lib/refcount.c
+++ b/lib/refcount.c
@@ -33,6 +33,9 @@
  * Note that the allocator is responsible for ordering things between free()
  * and alloc().
  *
+ * The decrements dec_and_test() and sub_and_test() also provide acquire
+ * ordering on success.
+ *
  */
 
 #include <linux/mutex.h>
@@ -164,8 +167,8 @@ void refcount_inc_checked(refcount_t *r)
  * at UINT_MAX.
  *
  * Provides release memory ordering, such that prior loads and stores are done
- * before, and provides a control dependency such that free() must come after.
- * See the comment on top.
+ * before, and provides an acquire ordering on success such that free()
+ * must come after.
  *
  * Use of this function is not recommended for the normal reference counting
  * use case in which references are taken and released one at a time.  In these
@@ -190,7 +193,12 @@ bool refcount_sub_and_test_checked(unsigned int i, refcount_t *r)
 
 	} while (!atomic_try_cmpxchg_release(&r->refs, &val, new));
 
-	return !new;
+	if (!new) {
+		smp_acquire__after_ctrl_dep();
+		return true;
+	}
+	return false;
+
 }
 EXPORT_SYMBOL(refcount_sub_and_test_checked);
 
@@ -202,8 +210,8 @@ bool refcount_sub_and_test_checked(unsigned int i, refcount_t *r)
  * decrement when saturated at UINT_MAX.
  *
  * Provides release memory ordering, such that prior loads and stores are done
- * before, and provides a control dependency such that free() must come after.
- * See the comment on top.
+ * before, and provides an acquire ordering on success such that free()
+ * must come after.
  *
  * Return: true if the resulting refcount is 0, false otherwise
  */
-- 
1.8.3.1